Method and apparatus for encoding and decoding a region enhancement layer - Patents.com

Abstract

The present application provides a method and apparatus for encoding and decoding a region enhancement layer. The method for encoding a region enhancement layer in the present application includes: obtaining reconstructed pixels of a base layer of a target region; inputting the reconstructed pixels into a correction network to obtain correction information of the target region; inputting the correction information and original pixels of the target region into an encoding network to obtain a residual feature map of an enhancement layer of the target region; and encoding the residual feature map to obtain an enhancement layer bitstream of the target region. The present application can improve the encoding efficiency and encoding accuracy of the enhancement layer.

Description

[Technical field]

[0001] This application relates to video encoding and decoding techniques, and in particular to a method and apparatus for encoding and decoding a region enhancement layer.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Chinese Patent Application No. 202210365196.9, filed with the China National Intellectual Property Office on April 8, 2022, and entitled "METHOD AND APPARATUS FOR ENCODING AND DECODING REGION ENHANCEMENT LAYER," which is incorporated herein by reference in its entirety. [Background technology]

[0003] In the video transmission process, especially in multi-user scenarios, requirements for adaptive bit rate adjustment and control for different users are proposed, since different users are in different network bandwidths and their devices have different processing capabilities. Layered video coding is also called scalable video coding. The concept of time domain, spatial domain and quality layering is put forward, and enhancement layer information is added based on the base layer. As a result, higher frame rate, higher resolution and higher quality video content can be obtained than the base layer. Different users can choose whether to use the enhancement layer bit stream to match the network bandwidth and device processing capabilities.

[0004] However, scalable video coding still suffers from problems of high complexity and low encoding and decoding efficiency. Summary of the Invention

[0005] The present application provides a method and apparatus for encoding and decoding a region enhancement layer so as to improve the encoding efficiency and encoding accuracy of the enhancement layer.

[0006] According to a first aspect, the present application provides a method for encoding a regional enhancement layer, the method includes the steps of obtaining reconstructed pixels of a base layer of a target region, inputting the reconstructed pixels into a correction network, obtaining correction information of the target region, inputting the correction information and original pixels of the target region into an encoding network, obtaining an enhancement layer residual feature map of the target region, and encoding the residual feature map to obtain an enhancement layer bitstream of the target region.

[0007] In this embodiment of the present application, the correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then encodes the residual feature map of the enhancement layer of the target region based on the correction information. The encoding of the enhancement layer is performed only for the necessary region (target region). This can reduce the encoding complexity of the enhancement layer and improve the encoding efficiency of the enhancement layer. In addition, the encoding is performed based on the correction information. This can improve the encoding accuracy.

[0008] The target area is intended to represent the location of the image block that is focused and processed in one encoding process in the solution in this embodiment of the present application. The shape of the target area may be a regular rectangle or square, or may be an irregular shape. This is not particularly limited in this specification.

[0009] In the encoding process, an initially obtained image block may be called an original block, and a pixel included in the image block may be called an original pixel. A reconstructed image block may be called a reconstructed block, and a pixel included in the reconstructed image block may be called a reconstructed pixel. However, in scalable video coding, different layers have different resolutions, frame rates, or qualities, but the encoding process is almost similar, and in particular, each layer includes an initial image block and a reconstructed image block.

[0010] Therefore, in this embodiment of the present application, there may be the following concepts for the target area:

[0011] At the base layer, the pixels contained in the originally obtained region are referred to as the original pixels of the region's base layer, and the pixels contained in the reconstructed region are referred to as the reconstructed pixels of the region's base layer.

[0012] In the enhancement layer, the pixels contained in the originally obtained region are called the original pixels of the enhancement layer of the region, and the pixels contained in the reconstructed region are called the reconstructed pixels of the enhancement layer of the region.

[0013] It should be noted that the concept of pixel quantity is not mentioned in the above description of original pixels or reconstructed pixels. However, it should be understood that a region often includes multiple pixels. This is not particularly limited in the embodiments of the present application.

[0014] In a possible implementation, the reconstructed pixels are input into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values, where the correction information is a plurality of pixel values ​​or a plurality of feature values.

[0015] In this embodiment of the application, the correction network may be implemented by using a neural network. For example, the correction network is constructed by using a neural network that cascades four convolutional / deconvolutional layers and three activation layers with interpolation. The input to the correction network is the reconstructed pixels of the base layer of the target region, and the output is the correction information corresponding to the target region. The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The size of the convolution kernel of each convolution layer may be set to 3×3, the number of channels for outputting the feature map is set to M, and downsampling by a factor of 2 is performed on the width and height of each convolution layer. It should be understood that the above example does not constitute a specific limitation. The size of the convolution kernel, the number of channels of the feature map, the downsampling factor, the number of downsamplings, the number of convolutional layers, and the number of activation layers may all be adjusted. This is not particularly limited in the embodiment of the application.

[0016] In this embodiment of the present application, multiple probability distributions may be obtained, where the multiple probability distributions correspond to multiple feature values ​​included in the residual feature map, and then entropy encoding is performed on the corresponding feature values ​​in the residual feature map based on the multiple probability distributions respectively to obtain an enhancement layer bitstream.

[0017] The residual feature map of the enhancement layer of the target region contains multiple feature values. In order to perform entropy coding on the multiple feature values, the probability distribution of the multiple feature values ​​needs to be obtained. Below are some methods to obtain the probability distribution.

[0018] 1. Obtain multiple probability distributions based on the correction information.

[0019] 1.1. Input the correction information into a probability estimation network to obtain multiple probability distributions.

[0020] The probability estimation network may also include a convolution layer and a GDN. There is no limitation on whether the probability estimation network is a different activation function. In this embodiment of the present application, the number of convolution layers is not limited, and the size of the convolution kernel is also not limited. In this embodiment of the present application, a probability distribution model is first used for modeling, and then the correction information is input into the probability estimation network to obtain model parameters, and the model parameters are substituted into the probability distribution model to obtain a probability distribution. The probability distribution model may be a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model. When the probability distribution is a Gaussian distribution, the model parameters are the values ​​of the mean parameter μ and the variance σ of the Gaussian distribution. When the probability distribution is a Laplacian distribution, the model parameters are the values ​​of the location parameter μ and the scale parameter b of the Laplacian distribution. It should be understood that in addition to the above-mentioned probability distribution models, another model may be alternatively used. This is not particularly limited in the present specification.

[0021] 1.2. Obtain multiple probability distributions based on the correction information of the residual feature map and the reconstructed side information.

[0022] In this embodiment of the present application, the residual feature map of the enhancement layer of the target region may be input into a side information extraction network to obtain side information of the residual feature map, entropy coding is performed on the side information, and the encoded side information is written into the bitstream, in which case the side information of the residual feature map is used as the reconstructed side information of the residual feature map.

[0023] a. If the correction information is multiple feature values, input the reconstructed side information into a side information processing network to obtain a first feature map, and then input the multiple feature values ​​and the first feature map into a probability estimation network to obtain multiple probability distributions.

[0024] b. If the correction information is multiple pixel values, the multiple pixel values ​​are input (estimated) into a feature estimation network to obtain a second feature map, the reconstructed side information is input into a side information processing network to obtain a first feature map, and the first feature map and the second feature map are input into a probability estimation network to obtain multiple probability distributions.

[0025] 2. Obtain multiple probability distributions based on the reconstructed side information of the residual feature maps.

[0026] 2.1. The reconstructed side information is input into a probability estimation network to obtain multiple probability distributions.

[0027] 2.2. Obtain multiple probability distributions based on the reconstructed side information and the reconstructed pixels.

[0028] In this embodiment of the present invention, the reconstructed pixels of the residual feature map may be input into a feature estimation network to obtain a third feature map, the reconstructed side information may be input into a side information processing network to obtain a first feature map, and the first feature map and the third feature map may be input into a probability estimation network to obtain a plurality of probability distributions.

[0029] In a possible implementation, the method further includes inputting the residual feature map into a side information extraction network to obtain side information of the residual feature map, performing entropy coding on the side information or the quantized side information, and writing the encoded side information into a bitstream.

[0030] In a possible implementation, the encoding network includes a first encoding network. And the step of inputting the correction information and the original pixels of the target region into the encoding network to obtain the residual feature map of the enhancement layer of the target region includes the steps of performing subtraction on the original pixels and corresponding pixel values ​​in the correction information when the correction information is a plurality of pixel values, and inputting the subtraction results into the first encoding network to obtain the residual feature map. For example, the corresponding may be understood as positional corresponding, that is, the subtraction is performed on the pixel values ​​of the original pixels and the correction information at the corresponding positions.

[0031] In a possible implementation, the encoding network includes a second encoding network. And the step of inputting the correction information and the original pixels of the target region into the encoding network to obtain the residual feature map of the enhancement layer of the target region includes the steps of inputting the original pixels into the second encoding network, and when the correction information is a plurality of feature values, performing subtraction on the output of any convolution layer in the second encoding network and the corresponding feature values ​​in the correction information, and inputting the subtraction result into a network layer after any convolution layer in the second encoding network to obtain the residual feature map. For example, the corresponding may be understood as positional corresponding. That is, the subtraction is performed on the feature values ​​of the corresponding positions in the output of any convolution layer in the second encoding network and the correction information.

[0032] As mentioned above, the correction information may have two cases. One is a plurality of pixel values, and the other is a plurality of feature quantities. Correspondingly, the encoding network may also use two structures. In this embodiment of the present application, the input to the encoding network (Encoder) on the encoder side is the correction information and the original pixels of the target region, and the output is the residual feature map of the enhancement layer of the target region. It should be noted that in addition to the above two examples of the encoding network, an encoding network of another structure may alternatively be used in the embodiment of the present application. This is not particularly limited in this specification.

[0033] In a possible implementation, the step of obtaining the reconstructed pixels of the base layer of the target region may include the step of encoding the image to which the target region belongs to obtain the base layer bitstream of the image, the step of decoding the base layer bitstream to obtain the reconstructed image of the base layer of the image, and the step of determining at least one region to be emphasized based on the reconstructed image, where the target region is one of the at least one region.

[0034] The step of determining at least one region to be emphasized based on the reconstructed image includes the step of dividing the reconstructed image to obtain a plurality of regions, and the step of determining, as the at least one region, a region whose variance is greater than a first threshold among the plurality of regions, or determining, for each of the plurality of regions, the ratio of pixels whose gradient is greater than a second threshold among all the pixels in the region, and then determining, as the at least one region, a region whose ratio is greater than a third threshold. For example, when the variance of a region is greater than a threshold t1 and t1 > 0, it may be considered that the texture of the region is complex, and thus it may be necessary to perform an enhancement process to improve the image quality. Or, the gradient is greater than a second threshold and the ratio of the pixels among all the pixels in the region is greater than a third threshold. For example, when the gradient is greater than a threshold a and the ratio of the pixels existing in all the pixels in the region is greater than a threshold t2, where a > 0 and 0 < t2 < 1, it may be considered that the texture of the region is complex, and thus it is necessary to perform an enhancement process to improve the image quality.

[0035] For the base layer, the encoder side encodes the original image to obtain a base layer bitstream, and then the base layer bitstream is decoded to obtain a reconstructed image of the base layer. For example, the VVC encoder encodes the original image to obtain a base layer bitstream, and the VVC decoder decodes the base layer bitstream to obtain a reconstructed image of the base layer. It should be understood that another encoder, HEVC codec, or AVC codec may be used for the base layer instead. This is not particularly limited in the embodiment of the present application.

[0036] In the above-mentioned multiple regions, based on the attributes or features of the image, not all regions need to be enhanced, and only the encoding and decoding of the base layer need to be performed for one or more regions to meet the image quality requirements. Therefore, in this embodiment of the present application, the target region is the region to be enhanced. In the subsequent encoding of the enhancement layer, the encoding and decoding of the enhancement layer can be performed only for the target region. In this way, the encoding and decoding efficiency of the image can be improved without the need to enhance the entire image.

[0037] In a possible implementation, the method further comprises the step of using side information of the residual feature map as reconstructed side information of the residual feature map.

[0038] According to a second aspect, the present application provides a method for decoding a regional enhancement layer, the method includes the steps of obtaining reconstructed pixels of a base layer of a target region, inputting the reconstructed pixels into a correction network, obtaining correction information of the target region, obtaining an enhancement layer bitstream of the target region, decoding the enhancement layer bitstream, obtaining a residual feature map of an enhancement layer of the target region, inputting the residual feature map and the correction information into a decoding network, obtaining reconstructed pixels of the enhancement layer of the target region.

[0039] In this embodiment of the present application, the correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then decodes the enhancement layer bitstream based on the correction information. The decoding of the enhancement layer is performed only for the necessary region (target region). This can reduce the decoding complexity of the enhancement layer and improve the decoding efficiency of the enhancement layer. In addition, the decoding is performed based on the correction information. This can improve the decoding accuracy.

[0040] In a possible implementation, the step of inputting the reconstructed pixels into a correction network to obtain correction information for the target area includes a step of inputting the reconstructed pixels into the correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​for the target area, where the correction information is the plurality of pixel values ​​or the plurality of feature values.

[0041] In a possible implementation, the step of decoding the enhancement layer bitstream to obtain a residual feature map of the enhancement layer for the target region includes the steps of: obtaining a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream; and performing entropy decoding on corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain the residual feature map.

[0042] In a possible implementation, obtaining a plurality of probability distributions based on the correction information includes inputting the correction information into a probability estimation network to obtain the plurality of probability distributions.

[0043] In a possible implementation, obtaining a plurality of probability distributions based on the correction information includes obtaining a plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map.

[0044] In a possible implementation, the step of obtaining the plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map includes, when the correction information is the plurality of feature values, inputting the reconstructed side information into a side information processing network to obtain a first feature map, and inputting the plurality of feature values ​​and the first feature map into a probability estimation network to obtain the plurality of probability distributions.

[0045] In a possible implementation, the step of obtaining a plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map includes, when the correction information is a plurality of pixel values, inputting the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, inputting the reconstructed side information into a side information processing network to obtain a first feature map, and inputting the first feature map and the second feature map into a probability estimation network to obtain the plurality of probability distributions.

[0046] In a possible implementation, the step of decoding the enhancement layer bitstream to obtain an enhancement layer residual feature map for the target region includes the steps of obtaining a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and performing entropy decoding on corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain the residual feature map.

[0047] In a possible implementation, the step of obtaining a plurality of probability distributions based on the reconstructed side information of the residual feature map includes inputting the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0048] In a possible implementation, obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map includes obtaining a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0049] In a possible implementation, the step of obtaining multiple probability distributions based on the reconstructed side information and the reconstructed pixels includes the steps of inputting the reconstructed pixels into a feature estimation network to obtain a third feature map, inputting the reconstructed side information into a side information processing network to obtain a first feature map, and inputting the first feature map and the third feature map into a probability estimation network to obtain multiple probability distributions.

[0050] In a possible implementation, the method further includes inputting the residual feature map into a side information extraction network to obtain side information of the residual feature map, and using the side information as reconstructed side information of the residual feature map.

[0051] In a possible implementation, the method further comprises the steps of obtaining a side information bitstream for the target region and parsing the side information bitstream to obtain the reconstructed side information.

[0052] In a possible implementation, the decoding network includes a first decoding network, and the step of inputting the residual feature map and the correction information into the decoding network to obtain reconstructed pixels of the enhancement layer of the target region includes: inputting the residual feature map into the first decoding network to obtain reconstructed residual pixels of the enhancement layer of the target region, and when the correction information is a plurality of pixel values, summing the reconstructed residual pixels and corresponding pixel values ​​in the correction information to obtain reconstructed pixels.

[0053] In a possible implementation, the decoding network includes a second decoding network, and the step of inputting the residual feature map and the correction information into the decoding network to obtain the reconstructed pixel of the enhancement layer of the target region includes: inputting the residual feature map into the second decoding network, and when the correction information is a plurality of feature values, adding the output of any convolution layer in the second decoding network and the corresponding feature value in the correction information, and inputting the addition result into a network layer after any convolution layer in the second decoding network to obtain the reconstructed pixel.

[0054] In a possible implementation, the method further includes the steps of obtaining a base layer bitstream of the image to which the target region belongs, parsing the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determining, based on the reconstructed image, at least one region to be enhanced, wherein the target region is one of the at least one region.

[0055] In a possible implementation, the step of determining at least one region to be enhanced based on the reconstructed image includes a step of dividing the reconstructed image to obtain a plurality of regions, and a step of determining as at least one region a region among the plurality of regions whose variance is greater than a first threshold, or a step of determining a proportion of pixels whose gradient is greater than a second threshold and is present in each of the plurality of regions out of all pixels, and a step of determining as at least one region a region whose proportion is greater than a third threshold.

[0056] According to a third aspect, the present application provides an encoding device, which includes: an acquisition module, configured to acquire reconstructed pixels of a base layer of a target region, a processing module, configured to input the reconstructed pixels to a correction network to acquire correction information of the target region, and input the correction information and original pixels of the target region to an encoding network to acquire an enhancement layer residual feature map of the target region, and an encoding module, configured to encode the residual feature map to acquire an enhancement layer bitstream of the target region.

[0057] In a possible implementation, the processing module is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area, where the correction information is the plurality of pixel values ​​or the plurality of feature values.

[0058] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0059] In a possible implementation, the encoding module is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0060] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the correction information of the residual feature map and the reconstructed side information.

[0061] In a possible implementation, the encoding module is specifically configured to input the reconstructed side information into a side information processing network to obtain a first feature map when the correction information is a plurality of feature values, and to input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0062] In a possible implementation, the encoding module is specifically configured, when the correction information is a plurality of pixel values, to input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0063] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0064] In a possible implementation, the encoding module is specifically configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0065] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0066] In a possible implementation, the encoding module is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0067] In a possible implementation, the encoding module is further configured to input the residual feature map into a side information extraction network to obtain side information of the residual feature map, and perform entropy coding on the side information or the quantized side information, and write the encoded side information into a bitstream.

[0068] In a possible implementation, the encoding network includes a first encoding network, and the encoding module is configured to perform subtraction on corresponding pixel values ​​in the original pixel and the correction information when the correction information is a plurality of pixel values, and input the subtraction result to the first encoding network to obtain a residual feature map.

[0069] In a possible implementation, the encoding network includes a second encoding network, and the encoding module is configured to input original pixels to the second encoding network, and when the correction information is a plurality of feature values, perform subtraction on the output of any convolution layer in the second encoding network and the corresponding feature value in the correction information, and input the subtraction result to a network layer after the any convolution layer in the second encoding network to obtain a residual feature map.

[0070] In a possible implementation, the encoding module is further configured to encode the image to which the target region belongs to to obtain a base layer bitstream of the image, decode the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0071] In a possible implementation, the encoding module is configured to divide the reconstructed image to obtain a plurality of regions, and determine at least one region in the plurality of regions in which the variance is greater than a first threshold, or determine a proportion of pixels in each of the plurality of regions in which the gradient is greater than a second threshold and is present in the total pixels, and determine at least one region in which the proportion is greater than a third threshold.

[0072] In a possible implementation, the encoding module is further configured to use side information of the residual feature map as reconstructed side information of the residual feature map.

[0073] According to a fourth aspect, the present application provides a decoding device, including: an acquisition module configured to acquire reconstructed pixels of a base layer for a target region; a processing module configured to input the reconstructed pixels into a correction network to acquire correction information for the target region; and a decoding module configured to decode the enhancement layer bitstream to acquire a residual feature map of an enhancement layer for the target region, where the acquisition module is further configured to acquire an enhancement layer bitstream for the target region, and the processing module is further configured to input the residual feature map and the correction information into a decoding network to acquire reconstructed pixels of the enhancement layer for the target region.

[0074] In a possible implementation, the processing module is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target region, where the correction information is the plurality of pixel values ​​or the plurality of feature values.

[0075] In a possible implementation, the decoding module is specifically configured to obtain a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and to perform entropy decoding on corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain a residual feature map.

[0076] In a possible implementation, the decoding module is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0077] In a possible implementation, the decoding module is specifically configured to obtain the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map.

[0078] In a possible implementation, the decoding module is specifically configured to input the reconstructed side information into a side information processing network to obtain a first feature map when the correction information is a plurality of feature values, and to input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0079] In a possible implementation, the decoding module is specifically configured, when the correction information is a plurality of pixel values, to input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, to input the reconstructed side information into a side information processing network to obtain a first feature map, and to input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0080] In a possible implementation, the decoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and to perform entropy decoding on corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain the residual feature map.

[0081] In a possible implementation, the decoding module is specifically configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0082] In a possible implementation, the decoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0083] In a possible implementation, the decoding module is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0084] In a possible implementation, the decoding module is further configured to input the residual feature map to a side information extraction network to obtain side information of the residual feature map, and use the side information as reconstructed side information of the residual feature map.

[0085] In a possible implementation, the decoding module is further configured to obtain a side information bitstream for the target region and to parse the side information bitstream to obtain the reconstructed side information.

[0086] In a possible implementation, the decoding network includes a first decoding network, and the decoding module is specifically configured to input the residual feature map into the first decoding network to obtain reconstructed residual pixels of the enhancement layer of the target region, and when the correction information is the plurality of pixel values, sum the reconstructed residual pixels and corresponding pixel values ​​in the correction information to obtain the reconstructed pixels.

[0087] In a possible implementation, the decoding network includes a second decoding network, and the decoding module is specifically configured to input the residual feature map into the second decoding network, and when the correction information is a plurality of feature values, sum the output of any convolution layer in the second decoding network and the corresponding feature value in the correction information, and input the sum result into a network layer after the any convolution layer in the second decoding network to obtain a reconstructed pixel.

[0088] In a possible implementation, the decoding module is further configured to obtain a base layer bitstream of the image to which the target region belongs, parse the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0089] In a possible implementation, the decoding module is specifically configured to divide the reconstructed image to obtain a plurality of regions, and determine as at least one region a region in which the variance in the plurality of regions is greater than a first threshold, or determine a proportion of pixels in each of the plurality of regions whose gradient is greater than a second threshold and is present in the total pixels, and determine as at least one region a region in which the proportion is greater than a third threshold.

[0090] According to a fifth aspect, the present application provides an encoder, comprising one or more processors and a memory configured to store one or more programs, the one or more programs being executed by the one or more processors to enable the one or more processors to perform a method according to any one of the possible implementations of the first aspect.

[0091] According to a sixth aspect, the present application provides a decoder, comprising one or more processors and a memory configured to store one or more programs, the one or more programs being executed by the one or more processors enabling the one or more processors to perform a method according to any one of the possible implementations of the second aspect.

[0092] According to a seventh aspect, the present application provides a computer-readable storage medium comprising a computer program which, when executed on a computer, enables the computer to carry out the method according to any one of the possible implementations of the first and second aspects.

[0093] According to an eighth aspect, the present application provides a computer program product, the computer program product comprising instructions which, when executed on a computer or processor, enable the computer or processor to perform a method according to any one of the possible implementations of the first and second aspects. [Brief description of the drawings]

[0094] [Figure 1] FIG. 1 is an exemplary diagram of layers of scalable video coding according to the present application. [Figure 2A] FIG. 2A is a block diagram of an example coding system 10. [Figure 2B] FIG. 2B is an exemplary diagram of a video coding system 40. [Diagram 3] FIG. 3 is a diagram illustrating a video coding device 400 in accordance with one embodiment of the present application. [Figure 4] FIG. 4 is a simplified block diagram of an example of an apparatus 500 according to one embodiment. [Diagram 5] FIG. 5 is a diagram of an application scenario according to one embodiment of the present application. [Figure 6] FIG. 6 is a diagram of an application scenario according to one embodiment of the present application. [Figure 7] FIG. 7 is a diagram of an application scenario according to one embodiment of the present application. [Figure 8] FIG. 8 is a flow chart of a method process 800 for encoding a region enhancement layer according to one embodiment of the present application. [Figure 9A] FIG. 9a is an exemplary diagram of a correction network. [Figure 9B] FIG. 9b is an exemplary diagram of a correction network. [Figure 9C] FIG. 9c is an exemplary diagram of a correction network. [Figure 9D] FIG. 9d is an exemplary diagram of a correction network. [Figure 10A] FIG. 10a is an exemplary diagram of an encoding network. [Figure 10B] FIG. 10b is an exemplary diagram of an encoding network. [Figure 11A] FIG. 11a is an exemplary diagram of a probability estimation network. [Figure 11B] FIG. 11b is an exemplary diagram of a probability estimation network. [Figure 12] FIG. 12 is a flow chart of a method process 1200 for decoding a region enhancement layer according to one embodiment of the present application. [Figure 13A] FIG. 13a is an exemplary diagram of a decoding network. [Figure 13B] FIG. 13b is an exemplary diagram of a decoding network. [Figure 14]FIG. 14 is an exemplary diagram of the encoding and decoding process. [Figure 15] FIG. 15 is an exemplary diagram of the encoding and decoding process. [Figure 16] FIG. 16 is an exemplary diagram of the encoding and decoding process. [Figure 17] FIG. 17 is an exemplary diagram of the encoding and decoding process. [Figure 18] FIG. 18 is an exemplary diagram of the structure of an encoding device 1800 according to an embodiment of the present application. [Figure 19] FIG. 19 is an exemplary diagram of the structure of a decoding device 1900 according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0095] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the accompanying drawings in the present application. It is clear that the described embodiments are not all of the embodiments of the present application, but rather only a part thereof. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0096] In the specification, embodiments, claims, and accompanying drawings of this application, the terms "first", "second", etc. are merely intended to distinguish and describe, and should not be understood as indicating or implying a relative importance or sequence. In addition, the terms "include", "have", and any variations thereof are intended to cover a non-exclusive inclusion, e.g., to include a series of steps or units. For example, a method, system, product, or device is not necessarily limited to the steps or units expressly recited, and may include other steps or units not expressly recited or inherent to such a process, method, product, or device.

[0097] In this application, it should be understood that "at least one (item)" means one or more, and "a plurality of" means two or more. The term "and / or" is used to describe an association relationship between related objects and represents that three relationships may exist. For example, "A and / or B" can represent three cases: only A is present, only B is present, and both A and B are present, where A and B may be singular or plural. The character " / " generally indicates an "or" relationship between related objects. The phrase "at least one of the following items" or similar phrases refers to any combination of these items, including any combination of single items or multiple items. For example, at least one item (piece) of a, b, or c may indicate a, b, c, "a and b," "a and c," "b and c," or "a and b and c," where a, b, and c may be singular or plural.

[0098] TECHNICAL FIELD The embodiments of the present application relate to the application of neural networks. For ease of understanding, the following will first explain and describe relevant nouns or terms.

[0099] 1. Neural Networks

[0100] A neural network (NN) is a machine learning model. A neural network may contain neurons. A neuron takes as input x s and an operation unit with an intercept of 1. Here, the output of the operation unit may be:

number

[0101] s = 1, 2, …, n, n is a natural number greater than 1, and W s x s where a is the weight of the neuron and b is the bias of the neuron. f is the activation function of the neuron, and is configured to introduce a non-linear feature into the neural network to convert the input signal at the neuron into an output signal. The output signal of the activation function may be used as the input of the next convolutional layer. The activation function may be a non-linear function such as ReLU. A neural network is a network formed by connecting many single neurons. Specifically, the output of a neuron may be the input of another neuron. The input of each neuron may be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field may be a region containing several neurons.

[0102] 2. Multi-layer perceptron (MLP)

[0103] MLP is a simple deep neural network (DNN) (different layers are fully connected), and is also called multi-layer neural network. MLP can be understood as a neural network with multiple hidden layers. There is no special metric for "multiple" in this specification. DNN is divided based on the location of different layers, and neural networks in DNN can be divided into three types: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer. The layers are fully connected. Specifically, the i-th layer (i th The neurons in the (i+1)th layer are always th DNNs are connected to neurons in layers. Although DNNs look complicated, in reality, DNNs are not complicated in terms of the work done in each layer, and can be expressed simply as the following linear relationship:

number

number

number

number

number

number

number

[0104] 3. Convolutional Neural Networks

[0105] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. In the deep learning architecture, multi-layer learning is performed at different abstraction levels according to machine learning algorithms. As a deep learning architecture, the CNN is a feed-forward artificial neural network. Neurons in a feed-forward artificial neural network may respond to images input into the neural network. A convolutional neural network includes a feature extractor composed of a convolutional layer and a pooling layer. The feature extractor may be viewed as a filter. The convolution process may be viewed as using a trainable filter to perform convolution on an input image or a convolutional feature plane (feature map).

[0106] A convolution layer is a neuron layer in a convolutional neural network where a convolution operation is performed on an input signal. A convolution layer may include multiple convolution operators. A convolution operator is also called a kernel. In image processing, a convolution operator acts as a filter to extract specific information from an input image matrix. A convolution operator may be essentially a weight matrix, and the weight matrix is ​​usually predefined. In the process of performing a convolution operation on an image, a weight matrix is ​​usually used to process pixels with a granularity of one pixel (or two pixels depending on the value of stride) in the horizontal direction of the input image to extract specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as that of the input image. During the convolution operation, the weight matrix spans the entire depth of the input image. Thus, a single depth dimension convolution output is generated through convolution with a single weight matrix. However, in most cases, a single weight matrix is ​​not used, but multiple weight matrices with the same size (row x column), i.e., multiple homogeneous matrices are applied. The output of the weight matrices may be stacked to form the depth dimension of the convolved image. The dimension in this specification may be understood as being determined based on the above-mentioned "multiple". Different weight matrices may be used to extract different features from the image. For example, one weight matrix is ​​used to extract side information of the image, another weight matrix is ​​used to extract a specific color of the image, and yet another weight matrix is ​​used to blur unwanted noise in the image. The size (row x column) of the multiple weight matrices is the same. The size of the feature maps extracted from the multiple weight matrices with the same size is also the same, and then the multiple extracted feature maps with the same size are combined to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through a large amount of training in practical applications.Each weight matrix including the weight values ​​obtained through training may be used to extract information from the input image, so that the convolutional neural network performs correct prediction. When the convolutional neural network has multiple convolutional layers, a large amount of general features are usually extracted in the first convolutional layer. The general features are also called low-level features. As the depth of the convolutional neural network increases, the features extracted in the subsequent convolutional layers become more complex, such as high-level semantic features. Features with higher levels of semantics are more applicable to the problem to be solved.

[0107] The amount of training parameters often needs to be reduced. Therefore, pooling layers often need to be introduced periodically after convolution layers. One convolution layer may be followed by one pooling layer, or multiple convolution layers may be followed by one or more pooling layers. During image processing, pooling layers are only used to reduce the spatial size of an image. The pooling layer may include an average pooling operator and / or a maximum pooling operator to perform sampling on an input image to obtain an image with a small size. The average pooling operator may be used to calculate pixel values ​​in an image within a certain range to generate an average value. The average value is used as the average pooling result. The maximum pooling operator may be used to select a pixel with a maximum value within a certain range as the maximum pooling result. In addition, just as the size of the weight matrix in the convolution layer needs to be related to the size of the image, the operator in the pooling layer also needs to be related to the size of the image. The size of the image output after processing in the pooling layer is smaller than the size of the image input to the pooling layer. Each pixel in the image output from the pooling layer represents the average or maximum value of the corresponding sub-area of ​​the image input to the pooling layer.

[0108] After the processing performed in the convolutional / pooling layer, the convolutional neural network is not ready to output the required output information. As mentioned above, in the convolutional / pooling layer, only features are extracted and the resulting parameters are reduced from the input image. However, to generate the final output information (required class information or other related information), the convolutional neural network needs to use neural network layers to generate the output of one required class or a group of required classes. Therefore, the convolutional neural network layer may include multiple hidden layers. The parameters included in the multiple hidden layers may be obtained through pre-training based on the relevant training data of a specific task type. For example, the task type may include image recognition, image classification, and super-resolution image reconstruction.

[0109] Optionally, in the neural network layer, multiple hidden layers are followed by an output layer of the entire convolutional neural network. The output layer has a loss function similar to categorical cross-entropy, and the loss function is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, back propagation is started to update the weight values ​​and deviations of each of the above layers, and reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network and the ideal result by using the output layer.

[0110] 4. Recurrent Neural Networks

[0111] Recurrent neural networks (RNNs) are used to process sequence data. In traditional neural network models, the layers from the input layer to the hidden layer and the output layer are fully connected, and the nodes in each layer are not connected. Although such general neural networks solve many problems, they still cannot solve many other problems. For example, to predict the next word in a sentence, the previous word must often be used because the previous word and the next word in the sentence are not independent. The reason why RNNs are called recurrent neural networks is that the current output of a sequence is also related to the previous output of the sequence. A specific representation form is one in which the network remembers previous information and applies the previous information to the calculation of the current output. Specifically, the nodes in the hidden layer are connected, and the input of the hidden layer includes the output of the input layer and also includes the output of the hidden layer at the previous time point. Theoretically, RNNs can process sequence data of any length. Training an RNN is similar to training a traditional CNN or DNN. The error backpropagation algorithm is also used, but with the following differences: When an RNN is extended, parameters such as W of the RNN are shared. This is different from the traditional neural network described in the example above. Also, while using the gradient descent algorithm, the output at each step depends on the network at the current step, and also on the network state at several previous steps. The learning algorithm is called the backpropagation through time (BPTT) algorithm.

[0112] Why are recurrent neural networks still needed when convolutional neural networks are available? The reason is simple. Convolutional neural networks assume that elements are independent of each other, like cats and dogs, and that inputs and outputs are also independent. However, in the real world, multiple elements are interconnected. For example, stocks change over time. As another example, a person says, "I like to travel, and my favorite place is Yunnan. If I have a chance, I will go there." Here, people should know that the person is going to "Yunnan". Because people perform inference from context. But how does a machine do that? Then RNNs come into play. RNNs are intended to allow machines to remember, like humans. Therefore, the output of an RNN needs to depend on the current input information and historical memory information.

[0113] 5. Loss Function

[0114] In the process of training a deep neural network, the predicted value of the current network can be compared with the actually expected target value, since the output of the deep neural network is expected to be as close as possible to the actually expected predicted value. Then, the weight vector of each layer of the neural network is updated based on the difference between the predicted value and the target value (indeed, before the first update, there is usually an initialization process, specifically, parameters are pre-configured for all layers of the deep neural network). For example, if the predicted value of the network is large, the weight vector is adjusted to reduce the predicted value, and the adjustment is continuously performed until the deep neural network can predict the actually expected target value or a value close to the actually expected target value. Therefore, "how to obtain the difference between the predicted value and the target value through comparison" needs to be defined in advance. This is the loss function or the objective function. The loss function and the objective function are important equations that measure the difference between the predicted value and the target value. The loss function is used as an example. The higher the output value (loss) of the loss function, the larger the difference is. Training a deep neural network is therefore a process of minimizing the loss as much as possible.

[0115] 6. Backpropagation Algorithm

[0116] The convolutional neural network may correct the parameter values ​​in the initial super-resolution model in the training process according to the error backpropagation (BP) algorithm. As a result, the error loss associated with the reconstruction of the super-resolution model is smaller. Specifically, the input signal is forward transferred until an error loss occurs at the output, and the parameters of the initial super-resolution model are updated based on the error backpropagation information so that the error loss converges. The backpropagation algorithm is an error loss-centric backpropagation motion intended to obtain parameters, such as the weight matrix, of the optimal super-resolution model.

[0117] 7. Generative Adversarial Networks

[0118] Generative adversarial network (GAN) is a deep learning model. The model includes at least two modules: one module is a generative model, and the other module is a discriminative model. The two modules learn from each other through game learning to generate better outputs. Both the generative model and the discriminative model may be neural networks, and specifically, may be deep neural networks or convolutional neural networks. The basic principle of GAN is as follows. A GAN for generating pictures is used as an example, and it is assumed that there are two networks, namely G (generator) and D (discriminator). G is a network for generating pictures. G receives random noise z and generates a picture by using the noise. Here, the picture is denoted as G(z). D is a discriminator network used to determine whether the picture is “real” or not. The input parameter of D is x, where x represents a picture, and the output D(x) represents the probability that x is a real picture. A value of 1 for D(x) indicates that the picture is 130% real. A value of 0 for D(x) indicates that the picture cannot be real. In the process of training a generative adversarial network, the goal of the generative network G is to generate pictures that are as real as possible to deceive the discriminative network D, and the goal of the discriminative network D is to distinguish between pictures generated by G and real pictures as much as possible. In this way, a dynamic “gaming” process, specifically an “adversary” in a “generative adversarial network”, exists between G and D.The final game result is that, in an ideal state, G can generate an image G(z) that is difficult to distinguish from a real image, and D is difficult to judge whether the image generated by G is real or not. Specifically, D(G(z))=0.5. In this way, a good generative model G is obtained and can be used to generate pictures.

[0119] FIELD OF THE DISCLOSURE The embodiments of the present application relate to scalable video coding. For ease of understanding, the following will first explain and describe relevant nouns or terms.

[0120] Layered video coding, also called scalable video encoding and decoding, is an extension encoding standard of current video coding standards (commonly known as scalable video coding (SVC) which is an extension of advanced video coding (AVC) (H.264) or scalable high efficiency video coding (SHVC) which is an extension of high efficiency video coding (HEVC) (H.265)). Scalable video coding is mainly used to solve the problems of packet loss and delay jitter caused by real-time network bandwidth changes in real-time video transmission.

[0121] The basic structure in scalable video coding may be referred to as a layer. In scalable video coding techniques, space domain layering (resolution layering) is performed on the original image block, so that bitstreams of different resolution layers may be obtained. Resolution is the size of the image block in pixels. The resolution at the lower layer is lower and the resolution at the higher layer is equal to or higher than the resolution at the lower layer. Alternatively, time domain layering (frame rate layering) is performed on the original image block, so that bitstreams of different frame rate grades may be obtained. The frame rate may be the amount of image frames contained in a video per unit time. The frame rate at the lower layer is lower and the frame rate at the higher layer is equal to or higher than the frame rate at the lower layer. Alternatively, quality domain layering is performed on the original image block, so that bitstreams of different encoding quality grades may be obtained. The encoding quality may be video quality. The degree of image distortion in the lower layer is high, and the degree of image distortion in the higher layer is not larger than the degree of image distortion in the lower layer.

[0122] A layer called base layer is usually the bottom layer in scalable video coding. In spatial domain layering, base layer image blocks are encoded at the lowest resolution. In temporal domain layering, base layer image blocks are encoded at the lowest frame rate. In quality domain layering, base layer image blocks are encoded at the highest QP or lowest bit rate. That is, the base layer is the lowest quality layer in scalable video coding. A layer called enhancement layer is a layer above the base layer in scalable video coding, and may be divided into multiple enhancement layers from bottom to top. The lowest enhancement layer encodes the encoding information obtained by the base layer to obtain a combined bitstream, and has a higher encoding resolution, a higher frame rate, or a higher bit rate than the base layer. The higher enhancement layer may encode the encoding information of the lower enhancement layer to obtain a higher quality image block.

[0123] For example, FIG. 1 is an exemplary diagram of layers of scalable video coding according to the present application. As shown in FIG. 1, after an original image block is transmitted to a scalable encoder, the original image block may be layered into a base layer image block B and an enhancement layer image block (from E1 to En, n≧1) according to different encoding configurations. Then, a bitstream including a base layer bitstream and an enhancement layer bitstream is obtained through encoding. The base layer bitstream is generally a bitstream obtained from an image block with a minimum resolution, a minimum frame rate, or a minimum encoding quality parameter. The enhancement layer bitstream is a bitstream obtained by encoding an image block together based on a base layer with a high resolution, a high frame rate, or a high encoding quality parameter. As the amount of enhancement layer increases, the spatial domain layer, the temporal domain layer, or the quality layer for encoding also increases. When the encoder transmits a bitstream to a decoder, the transmission of the base layer bitstream is reserved first. When the network has margin, the bitstream at the higher layer is transmitted gradually. The decoder first receives and decodes the base layer bitstream, and then decodes the received enhancement layer bitstream from the lower layer to the higher layer, layer by layer, in the spatial domain, the time domain, or in the ascending order of quality. And then overlaps the decoded information of the higher layer with the reconstructed block of the lower layer to obtain a reconstructed block with a higher resolution, a higher frame rate, or a higher quality.

[0124] As mentioned above, each image of a video sequence is usually partitioned into a set of non-overlapping blocks, and encoding is usually performed at the block level. In other words, an encoder usually processes video at the block (image block) level, i.e., encodes video, for example, generates a prediction block through spatial (intra-frame) prediction and temporal (inter-frame) prediction, subtracts the prediction block from an image block (a block currently being / to be processed) to obtain a residual block, transforms the residual block, and quantizes the residual block in the transform domain to reduce the amount of data to be transmitted (compressed). The encoder further needs to perform inverse quantization and inverse transformation to obtain a reconstructed residual block, and then adds pixel values ​​of the reconstructed residual block to pixel values ​​of the prediction block to obtain a reconstructed block. The reconstructed block in the base layer is a reconstructed block obtained by performing the above-mentioned operations on a base layer image block obtained by layering an original image block.

[0125] The region in the following can be the largest coding unit (LCU) in the entire image, an image block obtained by dividing the entire image, a region of interest (ROI) in the entire image (i.e. a designated image region in the image to be processed), or a slice in the image.

[0126] With the excellent performance of deep learning in various fields, researchers have proposed a deep learning-based end-to-end image encoding solution. In combination with this solution, the related art provides a layered image encoding framework, where the base layer uses a codec based on versatile video coding (VVC), and the enhancement layer uses a codec based on artificial intelligence (AI). This solution is described as follows:

[0127] Encoder side:

[0128] For the base layer, a VVC encoder encodes an input image x to obtain a bitstream, and a VVC decoder decodes the bitstream to obtain a reconstructed image xc of the base layer.

[0129] For the enhancement layer, x and xc are input to an encoding network (encoder) to obtain the residual feature map (y) of the enhancement layer. On the branch, y is input to a side information extraction network to obtain the side information (z) of y, and z is the quantized side information (

number

number

number

number

[0130] Decoder side:

[0131] For the base layer, the VVC decoder parses the bitstream to obtain the base layer reconstructed image xc.

[0132] For the enhancement layer, the bitstream includes the decoded side information (z ^ ) and then reconstructed side information (z ~ ) to obtain z ^Inverse quantization is performed on z ~ is y ^ The decoded residual feature map y ^ Entropy decoding is performed on the bitstream based on the probability distribution to obtain the reconstructed residual feature map (y ~ ) to obtain y ^ Inverse quantization is performed on xc and y ~ is input to the decoding network (Decoder) to obtain the reconstructed image yc of the enhancement layer.

[0133] In the above solution, AI image encoding and decoding methods are used in the enhancement layer, and are better than traditional image encoding solutions in terms of objective quality measurements such as MS structural similarity (SSIM) and peak signal-to-noise ratio (PSNR). However, in the above solution, the network complexity of AI image encoding and decoding is high, and there is a high computational power requirement. As a result, the efficiency of encoding and decoding in the enhancement layer is low.

[0134] Therefore, one embodiment of the present application provides a method for encoding and decoding a region enhancement layer, so as to improve the encoding and decoding efficiency of the enhancement layer. The following describes systems and / or scenarios to which the solution in the embodiment of the present application can be applied.

[0135] 2A is a block diagram of an exemplary coding system 10. A video encoder 20 (or simply referred to as encoder 20) and a video decoder 30 (or simply referred to as decoder 30) in the coding system 10 may be configured to perform various exemplary solutions described in the embodiments of the present application.

[0136] 2A, coding system 10 includes a source device 12. Source device 12 is configured to provide encoded image data 21, e.g., an encoded image, to a destination device 14 for decoding the encoded image data 21.

[0137] The source device 12 includes an encoder 20 and may additionally, i.e. optionally, include an image source 16, a pre-processor (or pre-processing unit) 18, e.g. an image pre-processor, and a communications interface (or communications unit) 22.

[0138] Image source 16 may include or be any kind of image capture device, such as a camera for capturing real-world images, and / or any type of image generation device, such as a computer graphics processing unit for generating computer animated images, or any type of other device for obtaining and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory or storage device that stores any of the above mentioned images.

[0139] To distinguish the processing performed by the pre-processor (or pre-processing unit 18 ), the image (or image data) 17 is also referred to as the original image (or original image data) 17 .

[0140] The pre-processor 18 is configured to receive the (original) image data 17 and perform pre-processing on the image data 17 to obtain a pre-processed image (or pre-processed image data) 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component.

[0141] Video encoder (or encoder) 20 is configured to receive pre-processed image data 19 and provide encoded image data 21 (further details are described below, e.g., with reference to FIG. 3).

[0142] The communications interface 22 of the source device 12 may be configured to receive the encoded image data 21 and to transmit the encoded image data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0143] Destination device 14 includes a decoder 30, and may additionally, that is, optionally, include a communications interface (or communications unit) 28, a post-processor (or post-processing unit) 32, and a display device 34.

[0144] The communications interface 28 of the destination device 14 is configured to receive the encoded image data 21 (or any further processed version thereof), e.g., directly from the source device 12 or from any other source device, e.g., a storage device, e.g., an encoded image data storage device, and to provide the encoded image data 21 to a decoder 30.

[0145] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded image data (or encoded data) 21 via a direct communication link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network, or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0146] The communications interface 22 may be configured, for example, to package the encoded image data 21 into an appropriate format, e.g., packets, and / or process the encoded image data using any type of transmission encoding or processing for transmission over a communications link or network.

[0147] Communications interface 28, forming a counterpart to communications interface 22, may be configured, for example, to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or depackaging to obtain encoded image data 21.

[0148] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in FIG. 2A, or as bidirectional communication interfaces, and may be configured, for example, to send and receive messages, for example, to set up connections, acknowledge and exchange any other information related to the communication link and / or data transmission, e.g., encoded image data transmission.

[0149] Video decoder (or decoder) 30 is configured to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (further details are described below, e.g., based on FIG. 4).

[0150] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), e.g., a decoded image, to obtain post-processed image data 33, e.g., a post-processed image. The post-processing performed by the post-processor 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, resampling, or any other processing to prepare the decoded image data 31 for display by, e.g., a display device 34.

[0151] The display device 34 is configured to receive the post-processed image data 33, for example, to display the image to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed image, such as, for example, an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0152] Coding system 10 further includes a training engine 25. Training engine 25 is configured to train encoder 20 or decoder 30, and in particular, the neural network (described in more detail below) used in encoder 20 or decoder 30.

[0153] In an embodiment of the present invention, the training data may be stored in a database (not shown), and the training engine 25 performs training based on the training data to obtain a neural network. It should be noted that the source of the training data is not limited in the embodiment of the present application. For example, the training data may be obtained from the cloud or other places to perform model learning.

[0154] 2A illustrates that source device 12 and destination device 14 are separate devices, an embodiment of the device may also include both source device 12 and destination device 14, or the functionality of both source device 12 and destination device 14, i.e., both source device 12 or corresponding functionality, and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality, and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any combination thereof.

[0155] Following this description, it will be apparent to those skilled in the art that the presence and (exact) division of different units or functions in the source device 12 and / or destination device 14 shown in FIG. 2A may vary depending on the actual device and application.

[0156] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented via processing circuitry as shown in FIG. 2B, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated processors for video coding, or any combination thereof. The encoder 20 and the decoder 30 may be implemented separately by using a processing circuitry 46. The processing circuitry 46 may be configured to perform various operations, as described below. When the technique is implemented partially in software, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium. Then, the device may execute the instructions in hardware using one or more processors to perform the technique of the present application. Either the encoder 20 and the decoder 30 may be integrated as part of a combined encoder / decoder (encoder / decoder, CODEC) in a single device, for example as shown in FIG. 2B.

[0157] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content delivery server), and the like, and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may include components for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0158] In some cases, the coding system 10 shown in FIG. 2A is merely an example, and the techniques of the present application may be applicable to video coding devices (e.g., video encoding or video decoding) that do not necessarily include data communication between the encoding device and the decoding device. In other examples, data may be retrieved from local memory, transmitted over a network, etc. A video encoding device may encode data and store the encoded data in memory, and / or a video decoding device may retrieve data from memory and decode the data. In some cases, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data into memory and / or retrieve data from memory and decode the data.

[0159] 2B is an example diagram of a video coding system 40. Video coding system 40 may include imaging device 41, video encoder 20, video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0160] 2B, imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 may be in communication with each other. Video coding system 40 may include only video encoder 20 or only video decoder 30 in different embodiments.

[0161] In some embodiments, antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Additionally, in some embodiments, display device 45 may be configured to present the video data. Processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, and the like. Video coding system 40 may also include optional processor 43. Optional processor 43 may also include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, and the like. In addition, memory 44 may be any type of memory, such as a volatile memory (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)) or a non-volatile memory (e.g., flash memory). In a non-limiting example, memory 44 may be implemented by a cache memory. In another example, processing circuitry 46 may include a memory (e.g., a cache) for implementing an image buffer.

[0162] In some examples, video encoder 20 implemented using logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented using processing circuitry 46. The logic circuitry may be configured to perform various operations described herein.

[0163] In some examples, video decoder 30 may be implemented by processing circuitry 46 in a similar manner to implement various modules described with reference to video decoder 30 in FIG. 4 and / or any other decoder system or subsystem described herein. In some embodiments, video decoder 30 implemented by using logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 implemented by using processing circuitry 46.

[0164] In some embodiments, antenna 42 may be configured to receive an encoded bitstream of video data. As described, the encoded bitstream may include data related to the video frame encoding described herein, indicators, index values, mode selection data, etc., such as data related to encoding partitioning (e.g., transform coefficients or quantized transform coefficients, optional indicators (as described), and / or data defining the encoding partitioning). Video coding system 40 may further include a video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0165] In embodiments of the present application, for examples described with reference to video encoder 20, it should be understood that video decoder 30 may be configured to perform the inverse process. With respect to signaling syntax elements, video decoder 30 may be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some embodiments, video encoder 20 may entropy encode the syntax elements into an encoded video bitstream. In such examples, video decoder 30 may parse such syntax elements and decode the associated video data accordingly.

[0166] For ease of explanation, embodiments of the present invention are described herein with reference to reference software for, for example, High-Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC), a next-generation video coding standard developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Collaboration Team on Video Coding (JCT-VC) of the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will appreciate that embodiments of the present invention are not limited to HEVC or VVC.

[0167] 3 is a diagram of a video coding device 400 according to one embodiment of the present invention. The video coding device 400 is suitable for implementing the embodiments described and disclosed herein. In one embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 in FIG. 2A, or an encoder, such as the video encoder 20 in FIG. 2A.

[0168] Video coding device 400 includes ingress ports 410 (or input ports 410) and receiver units (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430, e.g., processor 430 may be a neural network processing unit 430, for processing the data, transmitter units (Tx) 440 and egress ports 450 (or output ports 450) for transmitting data, and memory 460 for storing data. Video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to ingress ports 410, receiver units 420, transmitter units 440, and egress ports 450 for emission or ingress of optical or electrical signals.

[0169] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network NN-based coding module 470). The coding module 470 implements the above-disclosed embodiments. For example, the coding module 470 performs, processes, prepares, or provides various encoding operations. Thus, the coding module 470 provides substantial improvements to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0170] Memory 460 may include one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device, storing programs when such programs are selected for execution, and storing instructions and data read during program execution. Memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0171] 4 is a simplified block diagram of an apparatus 500 according to one embodiment, which may be used as either or both of the source device 12 and the destination device 14 in FIG.

[0172] The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices now existing or later developed that is capable of manipulating or processing information. Although the disclosed implementations may be implemented with a single processor, such as processor 502, as shown, advantages in speed and efficiency may be achieved using two or more processors.

[0173] In one implementation, the memory 504 in the device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable class of storage device may be used as the memory 504. The memory 504 may include code and data 506 that is accessed by the processor 502 via a bus 512. The memory 504 may further include an operating system 508 and application programs 510. The application programs 510 include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N and further include a video coding application that performs the methods described herein.

[0174] The apparatus 500 may include one or more output devices, such as a display 518. In one embodiment, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0175] Although bus 512 of device 500 is described herein as a single bus, bus 512 may include multiple buses. Additionally, secondary storage may be directly coupled to the other components of device 500 or may be accessed over a network, and may include multiple units, such as a single integrated unit, e.g., a memory card, or multiple memory cards. Thus, device 500 may have a wide variety of configurations.

[0176] 5 is a diagram of an application scenario according to one embodiment of the present application. As shown in FIG. 5, an application scenario may be a service related to image / video capture, storage, or transmission in a terminal, a cloud server, or video surveillance. For example, it may be photo taking / video recording by a terminal, an album, a cloud album, or video surveillance.

[0177] Encoder side: A camera captures an image / video. An artificial intelligence (AI) image / video coding network obtains low redundancy image features from feature extraction for the image / video, and then performs compression based on the image features to obtain a bitstream / image file.

[0178] Decoder side: When an image / video needs to be output, the AI ​​image / video decoding network decompresses the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain a reconstructed image / video.

[0179] The storage / transmission module stores (e.g., captured by a terminal, video surveillance, or cloud server) or transmits (e.g., cloud services or live broadcast technology) the bitstreams / image files obtained by compression for different services.

[0180] 6 is a diagram of an application scenario according to one embodiment of the present application. As shown in FIG. 6, the application scenario may be a service related to image / video capture, storage, or transmission in a terminal or video surveillance, such as terminal album, video surveillance, or live broadcast.

[0181] Encoder side: The encoding network converts the image / video into image features with lower redundancy. The encoding network usually includes a nonlinear transformation unit and has nonlinear characteristics. The entropy estimation network is responsible for calculating the encoding probability for each piece of data in the image features. The entropy coding network performs lossless encoding on the image features based on the probability corresponding to each piece of data to obtain a bitstream / image file. This further reduces the amount of data transmission in the image compression process.

[0182] Decoder side: The entropy decoding network performs lossless decoding on the bitstream / image file based on the probability corresponding to each piece of data to obtain reconstructed image features. The decoding network performs inverse transformation on the image features output through entropy decoding, and parses the image features into an image / video. Corresponding to the encoding network, the decoding network usually includes a nonlinear transformation unit and has nonlinear characteristics. The storage module saves the bitstream / image file in a corresponding storage location of the terminal. The loading module loads the bitstream / image file from the corresponding storage location of the terminal, and inputs the bitstream / image file into the entropy decoding network.

[0183] 7 is a diagram of an application scenario according to one embodiment of the present application. As shown in FIG. 7, the application scenario may be a service related to image / video capture, storage, or transmission in the cloud or video surveillance, such as cloud album, video surveillance, or live broadcast.

[0184] Encoder side: The local side gets the image / video, encodes the image / video (JPEG) to get the compressed image / video, and then sends the compressed image / video to the cloud. The cloud performs JPEG decoding on the compressed image / video to get the image / video, and then compresses the image / video to get the bitstream / image file, and stores the bitstream / image file.

[0185] Decoder side: When the local side needs to obtain an image / video from the cloud, the cloud decompresses the bitstream / image file to obtain the image / video, performs JPEG encoding on the image / video to obtain a compressed image / video, and then sends the compressed image / video to the local side. The local side performs JPEG decoding on the compressed image / video to obtain the image / video. For the structure of the cloud and the usage of each module, please refer to the structure and usage of each module in Figure 7. The details are not described in this specification in the embodiment of this application.

[0186] Based on the above encoding / decoding network and application scenarios, the embodiments of the present application provide an image encoding / decoding method to implement efficient nonlinear transform processing and improve the rate-distortion performance of image / video compression algorithms.

[0187] FIG. 8 is a flow chart of a process 800 of a method for encoding a region enhancement layer according to one embodiment of the present application. The process 800 may be performed by the video encoder 20. The process 800 is described as a series of steps or operations. It should be understood that the steps or operations of the process 800 may be performed in various orders and / or simultaneously, and are not limited to the order of execution shown in FIG. 8. Assuming that a video data stream having multiple image frames is used by the video encoder 20, the process 800 is performed to encode a region enhancement layer, the process 800 including the following steps:

[0188] Step 801: Obtain the reconstructed pixels of the base layer of the target region.

[0189] Usually, each image of a video sequence may be partitioned into a set of non-overlapping image blocks. And then encoding is performed at the image block level. In other words, an encoder usually processes video at the block (image block) level, i.e., encodes video, for example, generates a predictive block through spatial (intra-frame) prediction and temporal (inter-frame) prediction, subtracts the predictive block from an image block (a block currently being / to be processed) to obtain a residual block, transforms the residual block, and quantizes the residual block in the transform domain to reduce the amount of data to be transmitted (compressed). The encoder further needs to perform inverse quantization and inverse transformation to obtain a reconstructed residual block, and then adds pixel values ​​of the reconstructed residual block to pixel values ​​of the predictive block to obtain a reconstructed block. In this embodiment of the present application, the region included in the image frame may be a largest coding unit (LCU) in the entire image, an image block obtained by dividing the entire image, a region of interest (ROI) in the entire image (i.e., a designated image region to be processed in the image), etc. In addition to the several cases mentioned above, it should be understood that a region may alternatively be a sub-image described in another manner, which is not specifically limited herein.

[0190] Based on this, the target area is intended to represent the location of the image block that is focused and processed in one encoding process in the solution in this embodiment of the present application. The shape of the target area may be a regular rectangle or square, or may be an irregular shape. This is not particularly limited in this specification.

[0191] As mentioned above, in the encoding process, an initially obtained image block may be referred to as an original block, and a pixel included in the image block may be referred to as an original pixel. A reconstructed image block may be referred to as a reconstructed block, and a pixel included in the reconstructed image block may be referred to as a reconstructed pixel. However, in scalable video coding, different layers have different resolutions, frame rates, or qualities, but the encoding process is generally similar, and in particular, each layer includes an initial image block and a reconstructed image block.

[0192] Therefore, in this embodiment of the present application, there may be the following concepts for the target area:

[0193] In the base layer, the pixels contained in the originally obtained region are referred to as the original pixels of the region's base layer, and the pixels contained in the reconstructed region are referred to as the reconstructed pixels of the region's base layer.

[0194] In the enhancement layer, the pixels contained in the originally obtained region are referred to as the original pixels of the enhancement layer of the region, and the pixels contained in the reconstructed region are referred to as the reconstructed pixels of the enhancement layer of the region.

[0195] It should be noted that the concept of quantity of pixels is not mentioned in the above description of original pixels or reconstructed pixels, however, it should be understood that a region often includes multiple pixels, which is not particularly limited in the embodiments of the present application.

[0196] In this embodiment of the present application, obtaining the reconstructed pixels of the base layer of the target region may include: encoding an image to which the target region belongs to obtain a base layer bitstream of the image, decoding the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determining at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0197] For the base layer, the encoder side encodes an original image to obtain a base layer bitstream, and then the base layer bitstream is decoded to obtain a reconstructed image of the base layer. For example, the VVC encoder encodes an original image to obtain a base layer bitstream, and the VVC decoder decodes the base layer bitstream to obtain a reconstructed image of the base layer. It should be understood that another encoder, HEVC codec, or AVC codec can be alternatively used for the base layer. This is not particularly limited in the embodiment of the present application.

[0198] After the reconstructed image of the base layer is obtained, the reconstructed image may be divided to obtain multiple regions. For multiple regions, refer to the relevant description of the above-mentioned regions, such as multiple LCUs, multiple image blocks, and multiple ROIs. It should be noted that a region may be a partial image described in multiple ways. Correspondingly, multiple ways may be used as the division method to obtain multiple regions. This is not particularly limited in the embodiment of the present application.

[0199] In the above-mentioned multiple regions, based on the attributes or features of the image, it is not necessary for all regions to be enhanced, and only the encoding and decoding of the base layer need to be performed on one or more regions to meet the image quality requirements. Therefore, in this embodiment of the present application, the target region is the region to be enhanced. In the subsequent encoding of the enhancement layer, the encoding and decoding of the enhancement layer can be performed only on the target region. In this way, it is not necessary to emphasize the entire image, and the encoding and decoding efficiency of the image can be improved.

[0200] At least one of the multiple regions can be determined as the target region. At least one region satisfies the following conditions. That is, the variance of the region is greater than the first threshold. For example, when the variance of the region is greater than the threshold t1 and t1 > 0, the texture of the region may be considered complex, and therefore, enhancement processing needs to be performed to improve the image quality. Or, the gradient is greater than the second threshold, and the pixels of the pixels among all the pixels in the region are greater than the third threshold. For example, when the gradient is greater than the threshold a, and the pixels of all the pixels in the region are greater than the threshold t2, a > 0, and 0 < t2 < 1, the texture of the region may also be considered complex, and therefore, enhancement processing needs to be performed to improve the image quality.

[0201] After the above process, the encoder side uses any one of at least one region as the current target region, and can extract the pixels at the position corresponding to the target region from the reconstructed image of the base layer, that is, the reconstructed pixels of the base layer of the target region.

[0202] Step 802: Input the reconstructed pixels into the correction network to obtain the correction information of the target region.

[0203] The input to the correction network is the reconstructed pixels of the base layer of the target region, and the output is the correction information corresponding to the target region. The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may be formed by a convolution layer (conv) and an activation layer (ReLU). There is no limitation on whether the correction network includes an activation layer (generalized partition normalization, GDN), and there is no limitation on whether there is another activation function. In this embodiment of the present application, the number of convolution layers is not limited, and the size of the convolution kernel is also not limited. For example, the convolution kernel may be 3×3, 5×5, or 7×7.

[0204] In a possible implementation, the reconstructed pixels of the base layer of the target region are input to a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target region, where the correction information may be the plurality of pixel values ​​or the plurality of feature values.

[0205] In this embodiment of the present application, the correction network may be implemented by using a neural network. For example, the correction network is constructed using a neural network formed by interpolating and cascading four convolutional layers / deconvolutional layers and three activation layers. The size of the convolution kernel of each convolutional layer may be set to 3×3, the number of channels for outputting the feature map is set to M, and downsampling by a factor of 2 is performed on the width and height of each convolutional layer. It should be understood that the above example does not constitute a specific limitation. The size of the convolution kernel, the number of channels of the feature map, the downsampling coefficient, the number of downsamplings, the number of convolutional layers, and the number of activation layers may all be adjusted. This is not particularly limited in the embodiment of the present application.

[0206] After processing by the above-mentioned correction network, a plurality of pixel values ​​and / or a plurality of feature values ​​may be output. It should be understood that the input to the correction network is a reconstructed pixel, and the reconstructed pixel is within the range of the target region. Therefore, even if the outputted plurality of pixel values ​​and / or plurality of feature values ​​are not in one-to-one correspondence with the pixels in the target region, the plurality of pixel values ​​and / or plurality of feature values ​​may still be considered to be within the range of the target region, i.e., the plurality of pixel values ​​and / or plurality of feature values ​​may correspond to the target region. For example, the correction information is an upsampled value of the reconstructed pixel of the base layer of the target region. The resolution of the reconstructed pixel of the base layer of the target region may be the same as or different from the resolution of the reconstructed pixel of the enhancement layer of the target region. This is not particularly limited in the embodiment of the present application.

[0207] FIG. 9a is an exemplary diagram of a correction network. As shown in FIG. 9a, the correction network includes 6 convolution layers and 5 activation layers ReLU, and the size of the convolution kernel is 3×3. The reconstructed pixels of the base layer of the target area are input to the correction network, and a plurality of pixel values ​​are output. The correction network can perform denoising on the reconstructed pixels to remove noise signals that are not useful for encoding the AI ​​enhancement layer, and obtain a plurality of pixel values.

[0208] Fig. 9b is an exemplary diagram of a correction network. As shown in Fig. 9b, the correction network includes 6 convolution layers and 5 activation layers ReLU, and the size of the convolution kernel is 3 × 3. The reconstructed pixels of the base layer of the target region are input to the correction network, and multiple feature values ​​are output.

[0209] Fig. 9c is an exemplary diagram of a correction network. As shown in Fig. 9c, the correction network includes 8 convolution layers and 6 activation layers ReLU, and the size of the convolution kernel is 3 x 3. The reconstructed pixels of the base layer of the target region are input into the correction network, and multiple pixel values ​​and multiple feature values ​​are output from different layers of the correction network.

[0210] FIG. 9d is an exemplary diagram of a correction network. As shown in FIG. 9d, the correction network includes 5 convolution layers, 1 deconvolution layer, and 5 activation layers ReLU, and the size of the convolution kernel is 3×3. The size of the convolution kernel of the deconvolution layer of the correction network may be set to 3×3, the number of channels for outputting the feature map is set to 48 (or may be another value, not limited in this specification), and upsampling by a factor of 2 is performed on the width and height of the deconvolution layer. The output channel of the last convolution layer is M. When M is 3, multiple pixel values ​​are output. When M is 48, multiple feature values ​​are output. In this case, the resolution of the correction information is twice the resolution of the reconstructed pixels of the base layer.

[0211] It should be noted that in addition to the above four examples of the correction network, in the embodiments of the present application, a correction network of another structure may alternatively be used, which is not particularly limited herein.

[0212] Step 803: Input the correction information and original pixels of the target region into the encoding network to obtain the enhancement layer residual feature map of the target region.

[0213] The encoder side can extract pixels at positions corresponding to the target region from the original image, i.e., the original pixels of the target region.

[0214] As mentioned above, the correction information can have two cases: multiple pixel values ​​and multiple features. Correspondingly, the encoding network can also use two structures. In this embodiment of the present application, the inputs to the encoding network (Encoder) on the encoder side are the correction information and the original pixels of the target region, and the output is the residual feature map of the enhancement layer of the target region.

[0215] Fig. 10a is an exemplary diagram of an encoding network. As shown in Fig. 10a, the encoding network can be a first encoding network including four convolution layers and three GDNs. Subtraction is performed on the original pixels and corresponding pixel values ​​in the correction information (multiple pixel values), and then the subtraction result is input into the first encoding network to obtain a residual feature map. In this embodiment, since the correction information (multiple pixel values) corresponds to a pixel region, subtraction may be performed directly on the correction information and the original pixels of the target region.

[0216] FIG. 10b is an exemplary diagram of an encoding network. As shown in FIG. 10b, the encoding network can be a second encoding network including four convolution layers and three GDNs. An original pixel is input to the second encoding network, and subtraction is performed on the output of any convolution layer (e.g., the second convolution layer) in the second encoding network and the corresponding feature value in the correction information (multiple feature values), and then the subtraction result is input to a network layer after any convolution layer (e.g., the second convolution layer) in the second encoding network to obtain a residual feature map. In this embodiment, since the correction information (multiple feature values) corresponds to a feature region, the original pixel needs to be input to the second encoding network first and then converted into the feature region. Then, subtraction is performed on the feature region and the multiple feature values.

[0217] In addition to the above two examples of encoding networks, it should be noted that in the embodiments of the present application, encoding networks of other structures may alternatively be used, which is not particularly limited in this specification.

[0218] Step 804: Encode the residual feature map to obtain an enhancement layer bitstream for the target region.

[0219] In this embodiment of the present application, multiple probability distributions may be obtained, where the multiple probability distributions correspond to multiple feature values ​​included in the residual feature map, and then entropy coding is performed on the corresponding feature values ​​in the residual feature map based on the multiple probability distributions respectively to obtain an enhancement layer bitstream.

[0220] The residual feature map of the enhancement layer of the target region contains multiple feature values. In order to perform entropy coding on the multiple feature values, the probability distribution of the multiple feature values ​​needs to be obtained. Below are some methods to obtain the multiple probability distributions.

[0221] 1. Obtain multiple probability distributions based on correction information

[0222] 1.1. Input the correction information into a probability estimation network to obtain multiple probability distributions.

[0223] The probability estimation network may also include a convolution layer and a GDN. There is no limitation on whether the probability estimation network is another activation function. In this embodiment of the present application, the number of convolution layers is not limited, and the size of the convolution kernel is also not limited. In this embodiment of the present application, a probability distribution model is first used for modeling, and then the correction information is input into the probability estimation network to obtain model parameters, and the model parameters are substituted into the probability distribution model to obtain a probability distribution. The probability distribution model may be a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model. When the probability distribution is a Gaussian distribution, the model parameters are the values ​​of the mean parameter μ and the variance σ of the Gaussian distribution. When the probability distribution is a Laplacian distribution, the model parameters are the values ​​of the location parameter μ and the scale parameter b of the Laplacian distribution. It should be understood that in addition to the above-mentioned probability distribution models, another model may be alternatively used. This is not particularly limited in the present specification.

[0224] 11a is an exemplary diagram of a probability estimation network. As shown in FIG. 11a, the probability estimation network includes four convolution layers and three GDNs, and the correction information input to the probability estimation network is a plurality of pixel values.

[0225] 11b is an exemplary diagram of a probability estimation network. As shown in FIG. 11b, the probability estimation network includes two convolution layers and two GDNs, and the correction information input to the probability estimation network is a plurality of feature values.

[0226] It should be noted that in addition to the above two examples of the probability estimation network, a probability estimation network of another structure may alternatively be used in the embodiments of the present application, which is not particularly limited herein.

[0227] 1.2. Obtain multiple probability distributions based on the correction information of the residual feature map and the reconstructed side information

[0228] a. If the correction information is multiple feature values, input the reconstructed side information into a side information processing network to obtain a first feature map, and then input the multiple feature values ​​and the first feature map into a probability estimation network to obtain multiple probability distributions.

[0229] In this embodiment of the present application, the residual feature map of the enhancement layer of the target region may be input into a side information extraction network to obtain side information of the residual feature map, entropy coding is performed on the side information, and the encoded side information is written into the bitstream. In this case, the side information of the residual feature map is used as the reconstructed side information of the residual feature map. The side information can be regarded as a feature map obtained by further performing feature extraction on the residual feature map of the enhancement layer of the target region and having the same dimension as the above-mentioned residual feature map. Thus, the side information extraction network is configured to further perform feature extraction on the residual feature map of the enhancement layer of the target region and obtain a feature map having the same dimension as the above-mentioned residual feature map.

[0230] The side information processing network can perform feature extraction on the side information and output a first feature map having the same resolution as the residual feature map. For example, in the side information processing network, a neural network formed by interpolating and cascading three deconvolution layers and two activation layers is used to implement the above-mentioned function.

[0231] For the probability estimation network, please refer to the above description, and the details will not be described again here.

[0232] b. If the correction information is multiple pixel values, input (estimate) the multiple pixel values ​​into a feature estimation network to obtain a second feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the second feature map into a probability estimation network to obtain multiple probability distributions.

[0233] The feature estimation network can convert pixel values ​​(multiple pixel values) represented by pixel regions into feature values ​​(second feature map) represented by feature regions. The feature estimation network can use the structure of the probability estimation network shown in Fig. 11b, and is different from the probability estimation network in the input process, output process, and training process. Due to the difference in the input process, output process, and training process, even if the network structure is the same (i.e., the layer structure included is the same), the feature estimation network can still be regarded as a different network and complete different functions.

[0234] For the side information processing network and the probability estimation network, please refer to the above description, and the details will not be described again in this specification.

[0235] 2. Obtain multiple probability distributions based on the reconstructed side information of the residual feature maps

[0236] 2.1. Input the reconstructed side information into a probability estimation network to obtain multiple probability distributions.

[0237] For the probability estimation network, please refer to the above description, and the details will not be described again here.

[0238] 2.2. Obtain multiple probability distributions based on the reconstructed side information and the reconstructed pixels

[0239] In this embodiment of the present invention, the reconstructed pixels of the residual feature map may be input into a feature estimation network to obtain a third feature map, the reconstructed side information may be input into a side information processing network to obtain a first feature map, and the first feature map and the third feature map may be input into a probability estimation network to obtain a plurality of probability distributions.

[0240] For the feature estimation network, the side information processing network, and the probability estimation network, please refer to the above description, and the details will not be described again in this specification.

[0241] In this embodiment of the present application, AI encoding is applied to the enhancement layer of a selected target region. Based on this, the correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then encodes the residual feature map of the enhancement layer of the target region based on the correction information. The encoding of the enhancement layer is performed only on the necessary region (target region). This can reduce the encoding complexity of the enhancement layer and improve the encoding efficiency of the enhancement layer. In addition, the encoding is performed based on the correction information. This can improve the encoding accuracy.

[0242] FIG. 12 is a flow chart of a process 1200 of a method for decoding a region enhancement layer according to one embodiment of the present application. The process 1200 may be performed by the video decoder 30. The process 1200 is described as a series of steps or operations. It should be understood that the steps or operations of the process 1200 may be performed in various orders and / or simultaneously, and are not limited to the order of execution shown in FIG. 12. Assuming that the bitstream is used by the video decoder 30, the process 1200 including the following steps is performed to decode the bitstream and obtain the reconstructed pixels of the region enhancement layer. The process 1200 may include the following steps:

[0243] Step 1201: Obtain the reconstructed pixels of the base layer of the target region.

[0244] Corresponding to the encoder side, the decoder side may receive a bitstream from the encoder side, which includes a base layer bitstream obtained after the encoder side encodes the original pixels of the image. Thus, the decoder side decodes the base layer bitstream to obtain a reconstructed image of the base layer of the image.

[0245] In this embodiment of the present application, the decoder side can obtain the reconstructed pixels of the base layer of the target region in the same manner as that of the encoder side. For details, see the description of step 801. The details will not be described again in this specification.

[0246] Step 1202: Input the reconstructed pixels into a correction network to obtain correction information for the target region.

[0247] For step 1202, please refer to the description of step 802. The details will not be described again in this specification.

[0248] Step 1203: Obtain the enhancement layer bitstream of the target region.

[0249] In step 804, the encoder side encodes the residual feature map to obtain the enhancement layer bitstream of the target region. Accordingly, the bitstream received by the decoder side further includes the enhancement layer bitstream of the target region.

[0250] Step 1204: Decode the enhancement layer bitstream to obtain an enhancement layer residual feature map of the target region.

[0251] In this embodiment of the present application, multiple probability distributions may be obtained, where the multiple probability distributions correspond to multiple feature value bitstreams included in the enhancement layer bitstream, and then entropy decoding is performed on the corresponding feature value bitstreams in the enhancement layer bitstream based on the multiple probability distributions to obtain a residual feature map of the target region.

[0252] The enhancement layer bitstream includes multiple feature value bitstreams. To perform entropy decoding on the multiple feature value bitstreams, the probability distributions of the multiple feature value bitstreams need to be obtained. For some methods for obtaining the multiple probability distributions, please refer to the description of step 804. The details will not be described again in this specification.

[0253] Step 1205: Input the residual feature map and the correction information into a decoding network to obtain the reconstructed pixels of the enhancement layer of the target region.

[0254] Corresponding to step 803, the encoder side obtains a residual feature map of the enhancement layer of the target region through an encoding network based on the inputted correction information and the original pixels of the target region, and the decoder side obtains, conversely, a reconstructed pixel of the enhancement layer of the target region through a decoding network based on the inputted residual feature map and correction information.

[0255] The correction information can have two cases: one is multiple pixel values, and the other is multiple feature values. Correspondingly, the decoding network can also use two structures. In this embodiment of the present application, the decoding network (Decoder) on the decoder side inputs the correction information and residual feature map of the enhancement layer of the target region, and outputs the reconstructed pixels of the enhancement layer of the target region.

[0256] Fig. 13a is an exemplary diagram of a decoding network. As shown in Fig. 13a, the decoding network can be a first decoding network including four convolution layers and three GDNs. The residual feature map is first input into the first decoding network to obtain the reconstructed residual pixel of the enhancement layer of the target region, and then the reconstructed residual pixel and the corresponding pixel value in the correction information are summed to obtain the reconstructed pixel, where the correction information is a plurality of pixel values. In this embodiment, since the correction information (a plurality of pixel values) corresponds to the pixel region, the correction information and the reconstructed residual pixel of the enhancement layer of the target region can be directly added.

[0257] FIG. 13b is an exemplary diagram of a decoding network. As shown in FIG. 13b, the decoding network can be a second decoding network that includes four convolution layers and three GDNs. The residual feature map is first input to the second decoding network, and the output of any convolution layer (e.g., the second convolution layer) in the second decoding network and the corresponding feature value (multiple feature values) in the correction information are summed. And then the summation result is input to the network layer after any convolution layer (e.g., the second convolution layer) in the second decoding network to obtain a reconstructed pixel. In this embodiment, since the correction information (multiple feature values) corresponds to a feature region, the residual feature map needs to be first input to the second decoding network to be converted into a feature region, and then the feature region and the multiple feature values ​​are summed.

[0258] It should be noted that in addition to the above two examples of the decoding network, a decoding network of another structure may alternatively be used in the embodiments of the present application, which is not particularly limited in this specification.

[0259] In this embodiment of the present application, AI decoding is applied to the enhancement layer bitstream. On this basis, the correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then decodes the enhancement layer bitstream based on the correction information. The enhancement layer decoding is performed only for the necessary region (target region). This can reduce the decoding complexity of the enhancement layer and improve the decoding efficiency of the enhancement layer. In addition, the decoding is performed based on the correction information. This can improve the decoding accuracy.

[0260] Below, some specific embodiments are provided to illustrate the solutions of the embodiments of the present application.

[0261] Example 1 An exemplary diagram of the encoding and decoding process is shown in Figure 14. As shown in Figure 14, the process in this embodiment is described as follows.

[0262] Encoder side:

[0263] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0264] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, such as the H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0265] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0266] The determining method may include selecting an area having complex texture, selecting an area that is interesting to the human eye, or randomly selecting one or more areas. Related techniques may be applied in this step. This is not particularly limited in the present specification.

[0267] For example, if the variance of a region is greater than a threshold t1, the region is a region with complex texture and can be used as a target region. Or, if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region is determined as a region with complex texture and can be used as a target region. t1 and a are numbers greater than 0, and t2 is a number between 0 and 1.

[0268] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0269] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information includes the above-mentioned multiple pixel regions or the above-mentioned multiple feature regions. For example, the following three methods may be used:

[0270] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9a, and the pixel values ​​are output as p.

[0271] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9b, and multiple feature values ​​are output as p.

[0272] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0273] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0274] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. If the correction information is a pixel value, the encoder 2 uses the structure shown in Fig. 10a. If the correction information is a feature value, the encoder 2 uses the structure shown in Fig. 10b.

[0275] It should be noted that the specific quantization method is not limited in the embodiment of the present application. Alternatively, the floating-point value in y may be truncated to obtain an integer, or the quantization feature value may be obtained by performing quantization based on a preset quantization step.

[0276] 5. p is input to the probability estimation network, and y ^ The probability distribution of all feature values ​​of is output, and then, based on the probability distribution, ^Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0277] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be input to the probability estimation network, but the structure of the probability estimation network is diverse. When the correction information is pixel values, the probability estimation network uses the structure shown in FIG. 11a. When the correction information is feature values, the probability estimation network uses the structure shown in FIG. 11b.

[0278] Decoder side:

[0279] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain a reconstructed image xc.

[0280] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0281] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0282] 4. p is input to the probability estimation network, and y ^ A probability distribution of all feature values ​​of xc1 is output, and then entropy decoding (AD) is performed on the enhancement layer bitstream (Bitstream2) based on the probability distribution to obtain the enhancement layer residual feature map y ^ Then, get y ^ Inverse quantization (IQ) is performed on y to obtain the inverse quantized residual feature map (y ~ ) to get the

[0283] 5. p and y ~is input to the enhancement layer decoder (Decoder2), which outputs the reconstructed pixels (xd) of the enhancement layer of xc1.

[0284] 6. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0285] Example 2 Figure 15 is an exemplary diagram of the encoding and decoding process. As shown in Figure 15, the difference between this embodiment and embodiment 1 is in how to obtain the probability distribution.

[0286] Encoder side:

[0287] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0288] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, for example, H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0289] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0290] The determining method may include selecting an area having complex texture, selecting an area that is interesting to the human eye, or randomly selecting one or more areas. Related techniques may be applied in this step. This is not particularly limited in the present specification.

[0291] For example, if the variance of a region is greater than a threshold t1, the region is a region with complex texture and may be used as a target region, or if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region may be determined as a region with complex texture and may be used as a target region, where t1 and a are numbers greater than 0, and t2 is a number between 0 and 1.

[0292] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0293] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information is multiple pixel domains or multiple feature domains. For example, the following three methods may be used:

[0294] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9a, and the pixel values ​​are output as p.

[0295] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9b, and multiple feature values ​​are output as p.

[0296] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0297] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0298] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. If the correction information is a pixel value, the encoder 2 uses the structure shown in Fig. 10a. If the correction information is a feature value, the encoder 2 uses the structure shown in Fig. 10b.

[0299] It should be noted that the specific quantization method is not limited in the embodiment of the present application. Alternatively, the floating-point value in y may be truncated to obtain an integer, or the quantization feature value may be obtained by performing quantization based on a preset quantization step.

[0300] 5. y is input to the side information extraction network to obtain the side information z of y. Then, quantization (Q) is performed on z, and the quantized side information (z ^ ) and z ^ Entropy coding (AE) is performed on Bitstream 3 to obtain a bitstream (Bitstream 3), and then entropy decoding (AD) is performed on Bitstream 3 to obtain the reconstructed side information (z ^ In addition, the encoder side can send bitstream 3 to the decoder side.

[0301] 6.z ^ is input to the probability estimation network, and y ^ The probability distribution of all feature values ​​of is output, and then, based on the probability distribution, ^Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0302] Decoder side:

[0303] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain a reconstructed image xc.

[0304] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0305] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0306] 4. The bitstream (Bitstream3) is parsed and the reconstructed side information (z ^ ) to get the

[0307] 5.z ^ is input to the probability estimation network, which outputs the probability distribution of all feature bitstreams in the enhancement layer bitstream (Bitstream2), and then, based on the probability distribution, entropy coding (AE) is performed on Bitstream2 to obtain the enhancement layer residual feature map (y ^ ) to get the

[0308] 6. p and y ~ is input to the enhancement layer decoder (Decoder2), which outputs the reconstructed pixels (xd) of the enhancement layer of xc1.

[0309] 7. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0310] Example 3 Figure 16 is an exemplary diagram of the encoding and decoding process. As shown in Figure 16, the difference between this embodiment and embodiment 2 is in how to obtain the probability distribution.

[0311] Encoder side:

[0312] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0313] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, for example, H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0314] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0315] The determining method may include selecting an area having complex texture, selecting an area that is interesting to the human eye, or randomly selecting one or more areas. Related techniques may be applied in this step. This is not particularly limited in the present specification.

[0316] For example, if the variance of a region is greater than a threshold t1, the region is a region with complex texture and may be used as a target region, or if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region may be determined as a region with complex texture and may be used as a target region, where t1 and a are numbers greater than 0, and t2 is a number between 0 and 1.

[0317] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0318] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information is multiple pixel domains or multiple feature domains. For example, the following three methods may be used:

[0319] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9a, and the pixel values ​​are output as p.

[0320] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9b, and multiple feature values ​​are output as p.

[0321] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0322] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0323] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. If the correction information is a pixel value, the encoder 2 uses the structure shown in Fig. 10a. If the correction information is a feature value, the encoder 2 uses the structure shown in Fig. 10b.

[0324] It should be noted that the specific quantization method is not limited in the embodiment of the present application. Alternatively, the floating-point value in y may be truncated to obtain an integer, or the quantization feature value may be obtained by performing quantization based on a preset quantization step.

[0325] 5. y is input to the side information extraction network to obtain the side information z of y. Then, quantization (Q) is performed on z, and the quantized side information (z ^ ) and z ^ Entropy coding (AE) is performed on Bitstream 3 to obtain a bitstream (Bitstream 3), and then entropy decoding (AD) is performed on Bitstream 3 to obtain the reconstructed side information (z ^ In addition, the encoder side can send bitstream 3 to the decoder side.

[0326] 6.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0327] 7. If p is multiple pixel values, then multiple pixel values ​​need to be input to the feature estimation network for feature extraction. If p is multiple feature values, then multiple feature values ​​do not need to be input to the feature estimation network.

[0328] 8. The reconstructed side information of the feature domain and p of the feature domain are input to the probability estimation network, and y^ The probability distribution of all feature values ​​of y is output, and then, based on the probability distribution, ^ Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0329] Decoder side:

[0330] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain a reconstructed image xc.

[0331] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0332] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0333] 4. The bitstream (Bitstream3) is parsed and the reconstructed side information (z ^ ) to get the

[0334] 5.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0335] 6. If p is multiple pixel values, then the multiple pixel values ​​need to be input to the feature estimation network for feature extraction. If p is multiple feature values, then the multiple feature values ​​do not need to be input to the feature estimation network.

[0336] 7. The reconstructed side information of the feature domain and p of the feature domain are input into a probability estimation network, which outputs the probability distribution of all feature bitstreams in the enhancement layer bitstream (Bitstream2), and then, based on the probability distribution, entropy coding (AE) is performed on Bitstream2 to obtain the enhancement layer residual feature map (y ^ ) to get the

[0337] 8. p and y ~ is input to the enhancement layer decoder (Decoder2), which outputs the reconstructed pixels (xd) of the enhancement layer of xc1.

[0338] 9. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0339] Example 4 17 is an exemplary diagram of an encoding and decoding process. As shown in FIG. 17, the difference between this embodiment and embodiment 3 is in how to obtain the probability distribution.

[0340] Encoder side:

[0341] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0342] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, for example, H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0343] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0344] The determining method may include selecting an area having complex texture, selecting an area that is interesting to the human eye, or randomly selecting one or more areas. Related techniques may be applied in this step. This is not particularly limited in the present specification.

[0345] For example, if the variance of a region is greater than a threshold t1, the region is a region with complex texture and may be used as a target region, or if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region may be determined as a region with complex texture and may be used as a target region, where t1 and a are numbers greater than 0, and t2 is a number between 0 and 1.

[0346] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0347] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information is multiple pixel domains or multiple feature domains. For example, the following three methods may be used:

[0348] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9a, and the pixel values ​​are output as p.

[0349] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9b, and multiple feature values ​​are output as p.

[0350] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0351] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0352] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. If the correction information is a pixel value, the encoder 2 uses the structure shown in Fig. 10a. If the correction information is a feature value, the encoder 2 uses the structure shown in Fig. 10b.

[0353] It should be noted that the specific quantization method is not limited in the embodiment of the present application. Alternatively, the floating-point value in y may be truncated to obtain an integer, or the quantization feature value may be obtained by performing quantization based on a preset quantization step.

[0354] 5. y is input to the side information extraction network to obtain the side information z of y. Then, quantization (Q) is performed on z, and the quantized side information (z ^ ) and z ^Entropy coding (AE) is performed on Bitstream 3 to obtain a bitstream (Bitstream 3), and then entropy decoding (AD) is performed on Bitstream 3 to obtain the reconstructed side information (z ^ In addition, the encoder side can send bitstream 3 to the decoder side.

[0355] 6.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0356] 7.xc1 is input to the feature estimation network for feature extraction.

[0357] 8. The reconstructed side information and xc1, both of which belong to the feature domain, are input to a probability estimation network, y ^ The probability distribution of all feature values ​​of is output, and then, based on the probability distribution, ^ Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0358] Decoder side:

[0359] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain a reconstructed image xc.

[0360] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0361] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0362] 4. The bitstream (Bitstream3) is parsed and the reconstructed side information (z^ ) to get the

[0363] 5.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0364] 6.xc1 is input to the feature estimation network for feature extraction.

[0365] 7. The reconstructed side information and xc1, both of which belong to the feature domain, are input into a probability estimation network, which outputs the probability distribution of all feature bitstreams in the enhancement layer bitstream (Bitstream2), and then, based on the probability distribution, entropy coding (AE) is performed on Bitstream2 to obtain the enhancement layer residual feature map (y ^ ) to get the

[0366] 8. p and y ~ is input to the enhancement layer decoder (Decoder2), which outputs the reconstructed pixels (xd) of the enhancement layer of xc1.

[0367] 9. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0368] 18 is an exemplary diagram of a structure of an encoding device 1800 according to an embodiment of the present application. As shown in FIG. 18, the encoding device 1800 of this embodiment may be used on the encoder 20 side. The encoding device 1800 may include an acquisition module 1801, a processing module 1802, and an encoding module 1803.

[0369] The acquisition module 1801 is configured to acquire reconstructed pixels of a base layer of a target region. The processing module 1802 is configured to input the reconstructed pixels to a correction network to acquire correction information of the target region, and input the correction information of the target region and original pixels to an encoding network to acquire an enhancement layer residual feature map of the target region. The encoding module 1803 is configured to encode the residual feature map to acquire an enhancement layer bitstream of the target region.

[0370] In a possible implementation, the processing module 1802 is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area, where the correction information is the plurality of pixel values ​​or the plurality of feature values.

[0371] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0372] In a possible implementation, the encoding module 1803 is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0373] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map.

[0374] In a possible implementation, the encoding module 1803 is specifically configured to input the reconstructed side information into a side information processing network to obtain a first feature map when the correction information is a plurality of feature values, and to input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0375] In a possible implementation, the encoding module 1803 is specifically configured to, when the correction information is a plurality of pixel values, input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0376] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0377] In a possible implementation, the encoding module 1803 is configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0378] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0379] In a possible implementation, the encoding module 1803 is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0380] In a possible implementation, the encoding module 1803 is further configured to input the residual feature map into a side information extraction network to obtain side information of the residual feature map, and perform entropy coding on the side information or the quantized side information, and write the encoded side information into a bitstream.

[0381] In a possible implementation, the encoding network includes a first encoding network, and the encoding module 1803 is configured to perform subtraction on corresponding pixel values ​​in the original pixel and the correction information when the correction information is a plurality of pixel values, and input the subtraction result into the first encoding network to obtain a residual feature map.

[0382] In a possible implementation, the encoding network includes a second encoding network, and the encoding module 1803 is configured to input original pixels to the second encoding network, and when the correction information is a plurality of feature values, perform subtraction on the output of any convolutional layer in the second encoding network and the corresponding feature values ​​in the correction information, and input the subtraction result to a network layer after any convolutional layer in the second encoding network to obtain a residual feature map.

[0383] In a possible implementation, the encoding module 1803 is further configured to encode the image to which the target region belongs to obtain a base layer bitstream of the image, decode the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0384] In a possible implementation, the encoding module 1803 is specifically configured to divide the reconstructed image to obtain a plurality of regions, and determine at least one region in which the variance in the plurality of regions is greater than a first threshold, or determine a proportion of pixels in each of the plurality of regions whose gradient is greater than a second threshold and is present in the total pixels, and determine at least one region in which the proportion is greater than a third threshold.

[0385] In a possible implementation, the encoding module 1803 is further configured to use side information of the residual feature map as reconstructed side information of the residual feature map.

[0386] The apparatus in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 8, and the implementation principle and technical effect of the apparatus are similar and will not be described again in this specification.

[0387] 19 is an exemplary diagram of a structure of a decoding device 1900 according to an embodiment of the present application. As shown in FIG. 19, the decoding device 1900 in this embodiment can be used in the decoder side 30. The decoding device 1900 can include an acquisition module 1901, a processing module 1902, and a decoding module 1903.

[0388] The acquisition module 1901 is configured to acquire reconstructed pixels of a base layer for a target region. The processing module 1902 is configured to input the reconstructed pixels into a correction network to acquire correction information for the target region. The acquisition module 1901 is further configured to acquire an enhancement layer bitstream for the target region. The decoding module 1903 is configured to decode the enhancement layer bitstream to acquire an enhancement layer residual feature map for the target region. The processing module 1902 is further configured to input the residual feature map and the correction information into a decoding network to acquire an enhancement layer reconstructed pixels for the target region.

[0389] In a possible implementation, the processing module 1902 is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area, where the correction information is the plurality of pixel values ​​or the plurality of feature values.

[0390] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and to perform entropy decoding on the corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain a residual feature map.

[0391] In a possible implementation, the decoding module 1903 is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0392] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the correction information of the residual feature map and the reconstructed side information.

[0393] In a possible implementation, the decoding module 1903 is specifically configured to input the reconstructed side information into a side information processing network to obtain a first feature map when the correction information is a plurality of feature values, and to input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0394] In a possible implementation, the decoding module 1903 is specifically configured to, when the correction information is a plurality of pixel values, input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0395] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and to perform entropy decoding on corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain the residual feature map.

[0396] In a possible implementation, the decoding module 1903 is specifically configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0397] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0398] In a possible implementation, the decoding module 1903 is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0399] In a possible implementation, the decoding module 1903 is further configured to input the residual feature map into a side information extraction network to obtain side information of the residual feature map, and use the side information as reconstructed side information of the residual feature map.

[0400] In a possible implementation, the decoding module 1903 is further configured to obtain a side information bitstream for the target region and to parse the side information bitstream to obtain the reconstructed side information.

[0401] In a possible implementation, the decoding network includes a first decoding network, and the decoding module 1903 is specifically configured to input the residual feature map into the first decoding network to obtain a reconstructed residual pixel of the enhancement layer of the target region, and when the correction information is a plurality of pixel values, sum the reconstructed residual pixel and the corresponding pixel value in the correction information to obtain a reconstructed pixel.

[0402] In a possible implementation, the decoding network includes a second decoding network, and the decoding module 1903 is specifically configured to input the residual feature map into the second decoding network, and when the correction information is a plurality of feature values, sum the output of any convolutional layer in the second decoding network and the corresponding feature value in the correction information, and input the summation result into a network layer after the any convolutional layer in the second decoding network to obtain a reconstructed pixel.

[0403] In a possible implementation, the decoding module 1903 is further configured to obtain a base layer bitstream of an image to which the target region belongs, parse the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0404] In a possible implementation, the decoding module 1903 is specifically configured to divide the reconstructed image to obtain a plurality of regions, and determine as at least one region a region in which the variance in the plurality of regions is greater than a first threshold, or determine a proportion of pixels in each of the plurality of regions whose gradient is greater than a second threshold and is present in the total pixels, and determine as at least one region a region in which the proportion is greater than a third threshold.

[0405] The apparatus in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 12, and the implementation principle and technical effect of the apparatus are similar and will not be described again in this specification.

[0406] In the implementation process, the steps in the above-mentioned method embodiments can be implemented by using hardware integrated logic circuits in a processor or by using instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly presented as being performed and completed by a hardware encoding processor, or can be performed and completed by a combination of hardware modules and software modules in the encoding processor. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned method in combination with the processor's hardware.

[0407] The memory in the above-described embodiments may be volatile or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM) and is used as an external cache. By way of example and not limitation, many forms of RAM may be used, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchlink dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus random access memory (direct rambus RAM, DR RAM). It should be noted that memory in the systems and methods described herein includes, but is not limited to, these and any other suitable types of memory.

[0408] Those skilled in the art can recognize that in combination with the implementations described in the embodiments disclosed herein, the units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the functions described for each specific application, but the implementation should not be considered to go beyond the scope of this application.

[0409] Those skilled in the art will clearly understand that for convenience of description, the detailed operation processes of the above-mentioned systems, devices and units may be referred to the corresponding processes in the above-mentioned method embodiments, and the details will not be described again in this specification.

[0410] In some embodiments provided in the present application, it should be understood that the disclosed system, device, and method can be implemented in other manners. For example, the described device embodiment is merely an example. For example, the division into units is merely a logical functional division, and there may be other divisions in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. In addition, the shown or discussed mutual couplings, or direct couplings, or communication connections may be implemented by using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0411] The units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, located in one location or distributed over multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0412] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, and each of the units may exist physically alone, or two or more units may be integrated into one unit.

[0413] When the function is implemented in the form of a software functional unit and sold or used as an independent product, the function may be stored in a computer-readable storage medium. Based on such understanding, essentially, the technical solution in the present application, or a part of the contribution to the prior art, or a part of the technical solution, may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes some instructions for instructing a computer device (such as a personal computer, a server, a network device, etc.) to execute all or part of the steps of the method in the embodiments of the present application. The above-mentioned storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0414] The above description is merely a specific embodiment of the present application, and is not intended to limit the scope of protection of the present application. Any variation or replacement that is easily conceived by a person skilled in the art within the technical scope disclosed in the present application shall fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. 1. A method for encoding a region enhancement layer, comprising: obtaining reconstructed pixels of a base layer for a target region; inputting the reconstructed pixels into a correction network; obtaining correction information for the target area; inputting the correction information and original pixels of the target area into an encoding network; obtaining an enhancement layer residual feature map of the target region; encoding the residual feature map; obtaining an enhancement layer bitstream of the target region; and A method comprising:

2. inputting the reconstructed pixels into a correction network to obtain correction information for the target area, the step comprising: inputting the reconstructed pixels into the correction network; obtaining at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target region; the correction information is the plurality of pixel values ​​or the plurality of feature values; Step, The method of claim 1 , comprising:

3. encoding the residual feature map to obtain an enhancement layer bitstream for the target region, the step comprising: obtaining a plurality of probability distributions based on the correction information, the plurality of probability distributions corresponding to the plurality of feature values ​​included in the residual feature map; performing entropy coding on corresponding feature values ​​in the residual feature map based on each of the plurality of probability distributions to obtain the enhancement layer bitstream; The method of claim 1 or 2, comprising:

4. obtaining a plurality of probability distributions based on the correction information, inputting the correction information into a probability estimation network to obtain the plurality of probability distributions; The method of claim 3 , comprising:

5. obtaining a plurality of probability distributions based on the correction information, obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map; The method of claim 3 , comprising:

6. The step of obtaining the plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map includes: inputting the reconstructed side information into a side information processing network when the correction information is the plurality of feature values ​​to obtain a first feature map; inputting the plurality of feature values ​​and the first feature map into a probability estimation network to obtain the plurality of probability distributions; The method of claim 5 , comprising:

7. The step of obtaining the plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map includes: if the correction information is the pixel values, inputting the pixel values ​​into a feature estimation network to obtain a second feature map; inputting the reconstructed side information into a side information processing network to obtain a first feature map; inputting the first feature map and the second feature map into a probability estimation network to obtain the plurality of probability distributions; The method of claim 5 , comprising:

8. encoding the residual feature map to obtain an enhancement layer bitstream for the target region, the step comprising: obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map, the plurality of probability distributions corresponding to the plurality of feature values ​​included in the residual feature map; performing entropy coding on corresponding feature values ​​in the residual feature map based on each of the plurality of probability distributions to obtain the enhancement layer bitstream; The method of claim 1 or 2, comprising:

9. The step of obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map includes: inputting the reconstructed side information into a probability estimation network to obtain the plurality of probability distributions; The method of claim 8 , comprising:

10. The step of obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map includes: obtaining the plurality of probability distributions based on the reconstructed side information and the reconstructed pixels; The method of claim 8 , comprising:

11. The step of obtaining a plurality of probability distributions based on the reconstructed side information of the residual feature map and the reconstructed pixels includes: inputting the reconstructed pixels into a feature estimation network to obtain a third feature map; inputting the reconstructed side information into a side information processing network to obtain a first feature map; inputting the first feature map and the third feature map into a probability estimation network to obtain the plurality of probability distributions; The method of claim 10, comprising:

12. The method further comprises: inputting the residual feature map into a side information extraction network to obtain side information of the residual feature map; performing entropy coding on the side information or the quantized side information; writing the encoded side information into a bitstream; 12. The method of claim 5, comprising:

13. The encoding network includes a first encoding network; and The step of inputting the correction information and the original pixels of the target region into an encoding network to obtain an enhancement layer residual feature map of the target region includes: if the correction information is the plurality of pixel values, performing a subtraction on corresponding pixel values ​​in the original pixel and the correction information; inputting the subtraction result into the first encoding network to obtain the residual feature map; 13. The method of claim 1 , comprising:

14. the encoding network includes a second encoding network; and inputting the correction information and original pixels of the target region into an encoding network, and obtaining an enhancement layer residual feature map of the target region, inputting the original pixels into the second encoding network; if the correction information is the plurality of feature values, performing a subtraction on the output of any convolutional layer in the second encoding network and the corresponding feature value in the correction information; inputting the subtraction result into a network layer after the arbitrary convolutional layer in the second encoding network to obtain the residual feature map; 13. The method of claim 1 , comprising:

15. The method further comprises: encoding an image to which the target region belongs, obtaining a base layer bitstream of the image; decoding the base layer bitstream to obtain a reconstructed image of the base layer of the image; determining at least one region to be enhanced based on the reconstructed image, the target region being one of the at least one region; 15. The method of claim 1 , comprising:

16. The step of determining at least one region to be enhanced based on the reconstructed image comprises: Segmenting the reconstructed image to obtain a plurality of regions; determining, from among the plurality of regions, a region having a variance greater than a first threshold as the at least one region; or determining a percentage of all pixels in each of the plurality of regions whose gradient is greater than a second threshold, and determining the at least one region whose percentage is greater than a third threshold; 16. The method of claim 15, comprising:

17. the side information of the residual feature map is used as the reconstructed side information of the residual feature map.

13. The method according to any one of claims 5 to 12.

18. 1. A method for decoding a region enhancement layer, comprising: obtaining reconstructed pixels of a base layer for a target region; inputting the reconstructed pixels into a correction network to obtain correction information for the target area; obtaining an enhancement layer bitstream of the target region; decoding the enhancement layer bitstream to obtain an enhancement layer residual feature map of the target region; inputting the residual feature map and the correction information into a decoding network to obtain reconstructed pixels of the enhancement layer of the target region; A method comprising:

19. inputting the reconstructed pixels into a correction network, the step of obtaining correction information for the target area comprising: inputting the reconstructed pixels into the correction network; obtaining at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target region; the correction information is the plurality of pixel values ​​or the plurality of feature values; Step, 20. The method of claim 18, comprising:

20. Decoding the enhancement layer bitstream, wherein obtaining an enhancement layer residual feature map for the target region comprises: obtaining a plurality of probability distributions based on the correction information, the plurality of probability distributions corresponding to a plurality of feature value bitstreams included in the enhancement layer bitstream; performing entropy decoding on the corresponding feature value bitstream in the enhancement layer bitstream based on each of the plurality of probability distributions to obtain the residual feature map; 20. The method of claim 18 or 19, comprising:

21. The step of obtaining a plurality of probability distributions based on the correction information includes: inputting the correction information into a probability estimation network to obtain the plurality of probability distributions; 21. The method of claim 20, comprising:

22. The step of obtaining a plurality of probability distributions based on the correction information includes: obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map; 21. The method of claim 20, comprising:

23. The step of obtaining the plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map includes: inputting the reconstructed side information into a side information processing network when the correction information is the plurality of feature values ​​to obtain a first feature map; inputting the plurality of feature values ​​and the first feature map into a probability estimation network to obtain the plurality of probability distributions; 23. The method of claim 22, comprising:

24. The step of obtaining the plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map includes: if the correction information is the pixel values, inputting the pixel values ​​into a feature estimation network to obtain a second feature map; inputting the reconstructed side information into a side information processing network to obtain a first feature map; inputting the first feature map and the second feature map into a probability estimation network to obtain the plurality of probability distributions; 23. The method of claim 22, comprising:

25. a step of decoding the enhancement layer bitstream to obtain an enhancement layer residual feature map of the target region, the step comprising: obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map, the plurality of probability distributions corresponding to a plurality of feature value bitstreams included in the enhancement layer bitstream; performing entropy decoding on the corresponding feature value bitstream in the enhancement layer bitstream based on each of the plurality of probability distributions to obtain the residual feature map; 20. The method of claim 18 or 19, comprising:

26. The step of obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map includes: inputting the reconstructed side information into a probability estimation network to obtain the plurality of probability distributions; 26. The method of claim 25, comprising:

27. The step of obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map includes: obtaining the plurality of probability distributions based on the reconstructed side information and the reconstructed pixels; 26. The method of claim 25, comprising:

28. The step of obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map includes: inputting the reconstructed pixels into a feature estimation network to obtain a third feature map; inputting the reconstructed side information into a side information processing network to obtain a first feature map; inputting the first feature map and the third feature map into a probability estimation network to obtain the plurality of probability distributions; 28. The method of claim 27, comprising:

29. The method further comprises: inputting the residual feature map into a side information extraction network to obtain side information of the residual feature map; using the side information as the reconstructed side information of the residual feature map; 29. The method of any one of claims 22 to 28, comprising:

30. The method further comprises: obtaining a side information bitstream of the target region; parsing the side information bitstream to obtain the reconstructed side information; 29. The method of any one of claims 22 to 28, comprising:

31. The decoding network includes a first decoding network; and inputting the residual feature map and the correction information into a decoding network to obtain reconstructed pixels of the enhancement layer of the target region, the step comprising: inputting the residual feature map into the first decoding network to obtain reconstructed residual pixels of the enhancement layer of the target region; if the correction information is the plurality of pixel values, summing the reconstructed residual pixel with a corresponding pixel value in the correction information to obtain the reconstructed pixel; 31. The method of any one of claims 18 to 30, comprising:

32. the decoding network includes a second decoding network; and inputting the residual feature map and the correction information into a decoding network to obtain reconstructed pixels of the enhancement layer of the target region, the step comprising: inputting the residual feature map into the second decoding network; if the correction information is the plurality of feature values, summing an output of any convolutional layer in the second decoding network with a corresponding feature value in the correction information; inputting the summation result into a network layer after the arbitrary convolutional layer in the second decoding network to obtain the reconstructed pixel; 31. The method of any one of claims 18 to 30, comprising:

33. The method further comprises: obtaining a base layer bitstream of an image to which the target region belongs; parsing the base layer bitstream to obtain a reconstructed image of a base layer of the image; determining at least one region to be enhanced based on the reconstructed image, the target region being one of the at least one region; 33. The method of any one of claims 18 to 32, comprising:

34. The step of determining at least one region to be enhanced based on the reconstructed image comprises: Segmenting the reconstructed image to obtain a plurality of regions; determining, from among the plurality of regions, a region having a variance greater than a first threshold as the at least one region; or determining a percentage of pixels in each of the plurality of regions that have a gradient greater than a second threshold and that are present in the total pixels, and determining the at least one region as the region in which the percentage is greater than a third threshold; 34. The method of claim 33, comprising:

35. 1. An encoding device, comprising: an acquisition module configured to acquire reconstructed pixels of a base layer of a target region; and a processing module configured to input the reconstructed pixels into a correction network to obtain correction information for the target region, and input the correction information and original pixels of the target region into an encoding network to obtain an enhancement layer residual feature map for the target region; an encoding module configured to encode the residual feature map to obtain an enhancement layer bitstream for the target region; 13. An apparatus comprising:

36. The processing module is specifically configured to input the reconstructed pixels into the correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area; the correction information is the plurality of pixel values ​​or the plurality of feature values; 36. The apparatus of claim 35.

37. The encoding module is specifically configured to execute the method according to any one of claims 3 to 17.

37. Apparatus according to claim 35 or 36.

38. A decoding device, comprising: an acquisition module configured to acquire reconstructed pixels of a base layer for a target region; a processing module configured to input the reconstructed pixels into a correction network to obtain correction information for the target area; and a decoding module configured to decode the enhancement layer bitstream to obtain an enhancement layer residual feature map of the target region; The capture module is further configured to capture an enhancement layer bitstream of the target region; The processing module is further configured to input the residual feature map and the correction information into a decoding network to obtain reconstructed pixels of the enhancement layer of the target region. Device.

39. The processing module is specifically configured to input the reconstructed pixels into the correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area; the correction information is the plurality of pixel values ​​or the plurality of feature values; 39. The apparatus of claim 38.

40. The decoding module is configured to perform the method according to any one of claims 20 to 34.

40. Apparatus according to claim 38 or 39.

41. 1. An encoder comprising: one or more processors; a memory configured to store one or more programs; The one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method according to any one of claims 1 to 17. Encoder.

42. A decoder comprising: one or more processors; a memory configured to store one or more programs; The one or more programs, when executed by the one or more processors, enable the one or more processors to perform a method according to any one of claims 18 to 34. decoder.

43. A computer-readable storage medium having a computer program stored thereon, The computer program, when executed on a computer, enables the computer to carry out the method according to any one of claims 1 to 34. A computer-readable storage medium.

44. 1. A computer program product comprising instructions, The instructions, when executed on a computer or processor, enable the computer or processor to carry out a method according to any one of claims 1 to 34. Computer program products.

Images