Method and apparatus for encoding and decoding regional enhancement layers

Abstract

This application provides a method and an apparatus for encoding and decoding a region enhancement layer.SOLUTION: The method for encoding a region enhancement layer in the present application includes obtaining a reconstructed pixel of a base layer of a target region, inputting the reconstructed pixel into a correction network to obtain correction information of the target region, inputting the correction information and an original pixel of the target region into an encoding network to obtain a residual feature map of an enhancement layer of the target region, and encoding the residual feature map to obtain an enhancement layer bitstream of the target region. This application can improve encoding efficiency and encoding accuracy of an enhancement layer.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] This application relates to video encoding and decoding techniques, and in particular to a method and apparatus for encoding and decoding a region enhancement layer.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Chinese Patent Application No. 202210365196.9, filed with the State Intellectual Property Office of China on April 8, 2022, and entitled "METHOD AND APPARATUS FOR ENCODING AND DECODING REGION ENHANCEMENT LAYER," which is incorporated herein by reference in its entirety. [Background technology]

[0003] In the video transmission process, especially in multi-user scenarios, different users exist within different network bandwidths and their devices have different processing capabilities, which necessitates adaptive bitrate adjustment and control for different users. Layered video coding, also known as scalable video coding, proposes the concepts of temporal, spatial, and quality layering, and adds enhancement layer information based on the base layer. As a result, higher frame rates, higher resolutions, and higher-quality video content can be obtained than the base layer. Different users can choose whether to use the enhancement layer bitstream to match the network bandwidth and device processing capabilities.

[0004] However, scalable video coding still suffers from problems of high complexity and low encoding and decoding efficiency. Summary of the Invention

[0005] The present application provides a method and apparatus for encoding and decoding a regional enhancement layer so as to improve the encoding efficiency and encoding accuracy of the enhancement layer.

[0006] According to a first aspect, the present application provides a method for encoding a regional enhancement layer, the method including: obtaining reconstructed pixels of a base layer for a target region; inputting the reconstructed pixels into a correction network; obtaining correction information for the target region; inputting the correction information and original pixels of the target region into an encoding network; obtaining an enhancement layer residual feature map for the target region; and encoding the residual feature map; obtaining an enhancement layer bitstream for the target region.

[0007] In this embodiment of the present application, the correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then encodes the residual feature map of the enhancement layer for the target region based on the correction information. The encoding of the enhancement layer is performed only for the necessary region (target region), which can reduce the encoding complexity of the enhancement layer and improve the encoding efficiency of the enhancement layer. In addition, the encoding is performed based on the correction information, which can improve the encoding accuracy.

[0008] In the solution of this embodiment of the present application, the target area is intended to represent the location of an image block that is focused on and processed in one encoding process. The shape of the target area may be a regular rectangle or square, or may be an irregular shape. This is not particularly limited in this specification.

[0009] In the encoding process, an initially obtained image block may be referred to as an original block, and a pixel included in the image block may be referred to as an original pixel. A reconstructed image block may be referred to as a reconstructed block, and a pixel included in the reconstructed image block may be referred to as a reconstructed pixel. However, in scalable video coding, although different layers have different resolutions, frame rates, or qualities, the encoding process is generally similar, and in particular, each layer includes an initial image block and a reconstructed image block.

[0010] Therefore, in this embodiment of the present application, there may be the following concepts for the target area:

[0011] In the base layer, the pixels contained in the originally obtained region are referred to as the original pixels of the region's base layer, and the pixels contained in the reconstructed region are referred to as the reconstructed pixels of the region's base layer.

[0012] In the enhancement layer, the pixels contained in the originally obtained region are referred to as the original pixels of the enhancement layer of the region, and the pixels contained in the reconstructed region are referred to as the reconstructed pixels of the enhancement layer of the region.

[0013] It should be noted that the concept of pixel quantity is not mentioned in the above description of original pixels or reconstructed pixels. However, it should be understood that a region often includes multiple pixels. This is not particularly limited in the embodiments of the present application.

[0014] In a possible implementation, the reconstructed pixels are input to a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values, where the correction information is a plurality of pixel values ​​or a plurality of feature values.

[0015] In this embodiment of the present application, the correction network may be implemented using a neural network. For example, the correction network may be constructed using a cascaded neural network with four convolutional / deconvolutional layers and three activation layers interpolated. The input to the correction network is the reconstructed pixels of the base layer of the target region, and the output is correction information corresponding to the target region. The correction network functions to remove noise signals that are not useful for encoding in the AI ​​enhancement layer. The size of the convolution kernel of each convolutional layer may be set to 3x3, the number of channels for outputting the feature map is set to M, and downsampling by a factor of 2 is performed on the width and height of each convolutional layer. It should be understood that the above example does not constitute a specific limitation. The size of the convolutional kernel, the number of channels of the feature map, the downsampling factor, the number of downsamplings, the number of convolutional layers, and the number of activation layers may all be adjusted. This is not particularly limited in this embodiment of the present application.

[0016] In this embodiment of the present application, multiple probability distributions may be obtained, where the multiple probability distributions correspond to multiple feature values ​​included in the residual feature map, and then entropy encoding is performed on the corresponding feature values ​​in the residual feature map based on the multiple probability distributions respectively to obtain an enhancement layer bitstream.

[0017] The residual feature map of the enhancement layer for the target region contains multiple feature values. To perform entropy coding on the multiple feature values, the probability distribution of the multiple feature values ​​needs to be obtained. Below are some methods for obtaining the probability distribution:

[0018] 1. Obtain multiple probability distributions based on the correction information.

[0019] 1.1. Input the correction information into a probability estimation network to obtain multiple probability distributions.

[0020] The probability estimation network may also include a convolutional layer and a GDN. Whether the probability estimation network has a different activation function is not limited. In this embodiment of the present application, the number of convolutional layers is not limited, and the size of the convolution kernel is not limited. In this embodiment of the present application, a probability distribution model is first used for modeling, and then correction information is input into the probability estimation network to obtain model parameters. The model parameters are then substituted into the probability distribution model to obtain a probability distribution. The probability distribution model may be a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model. When the probability distribution is a Gaussian distribution, the model parameters are the values ​​of the mean parameter μ and variance σ of the Gaussian distribution. When the probability distribution is a Laplacian distribution, the model parameters are the values ​​of the location parameter μ and scale parameter b of the Laplacian distribution. It should be understood that in addition to the above-mentioned probability distribution models, other models may alternatively be used. This is not particularly limited in this specification.

[0021] 1.2. Obtain multiple probability distributions based on the correction information of the residual feature map and the reconstructed side information.

[0022] In this embodiment of the present application, the residual feature map of the enhancement layer of the target region may be input into a side information extraction network to obtain side information of the residual feature map, entropy coding is performed on the side information, and the encoded side information is written into the bitstream, in which case the side information of the residual feature map is used as the reconstructed side information of the residual feature map.

[0023] a. If the correction information is multiple feature values, input the reconstructed side information into a side information processing network to obtain a first feature map, and then input the multiple feature values ​​and the first feature map into a probability estimation network to obtain multiple probability distributions.

[0024] b. If the correction information is multiple pixel values, input (estimate) the multiple pixel values ​​into a feature estimation network to obtain a second feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and then input the first feature map and the second feature map into a probability estimation network to obtain multiple probability distributions.

[0025] 2. Obtain multiple probability distributions based on the reconstructed side information of the residual feature maps.

[0026] 2.1. The reconstructed side information is input into a probability estimation network to obtain multiple probability distributions.

[0027] 2.2. Obtain multiple probability distributions based on the reconstructed side information and the reconstructed pixels.

[0028] In this embodiment of the present invention, the reconstructed pixels of the residual feature map may be input into a feature estimation network to obtain a third feature map, the reconstructed side information may be input into a side information processing network to obtain a first feature map, and the first feature map and the third feature map may be input into a probability estimation network to obtain a plurality of probability distributions.

[0029] In a possible implementation, the method further includes inputting the residual feature map into a side information extraction network to obtain side information of the residual feature map; performing entropy coding on the side information or the quantized side information; and writing the encoded side information into a bitstream.

[0030] In a possible implementation, the encoding network includes a first encoding network. And, inputting the correction information and original pixels of the target region into the encoding network to obtain a residual feature map of the enhancement layer for the target region includes, when the correction information is a plurality of pixel values, performing subtraction on the original pixels and corresponding pixel values ​​in the correction information, and inputting the subtraction results into the first encoding network to obtain the residual feature map. For example, corresponding may be understood as positional corresponding, i.e., subtraction is performed on pixel values ​​at corresponding positions in the original pixels and the correction information.

[0031] In a possible implementation, the encoding network includes a second encoding network. The step of inputting the correction information and original pixels of the target region into the encoding network to obtain a residual feature map of the enhancement layer for the target region includes the steps of inputting the original pixels into the second encoding network, and, if the correction information is a plurality of feature values, performing subtraction on corresponding feature values ​​in the output of any convolutional layer in the second encoding network and the correction information, and inputting the subtraction result into a network layer after the any convolutional layer in the second encoding network to obtain the residual feature map. For example, the correspondence may be understood as positional correspondence. That is, the subtraction is performed on feature values ​​at corresponding positions in the output of any convolutional layer in the second encoding network and the correction information.

[0032] As mentioned above, the correction information can have two cases: one is a plurality of pixel values, and the other is a plurality of feature quantities. Correspondingly, the encoding network can also use two structures. In this embodiment of the present application, the input to the encoding network (Encoder) on the encoder side is the correction information and the original pixels of the target region, and the output is the residual feature map of the enhancement layer of the target region. It should be noted that in addition to the above two examples of the encoding network, encoding networks of other structures can alternatively be used in embodiments of the present application. This is not particularly limited in this specification.

[0033] In a possible implementation, the step of obtaining the reconstructed pixels of the base layer of the target region may include the steps of encoding the image to which the target region belongs to obtain the base layer bitstream of the image, decoding the base layer bitstream to obtain the reconstructed image of the base layer of the image, and determining at least one region to be emphasized based on the reconstructed image, where the target region is one of the at least one region.

[0034] The step of determining at least one region to be emphasized based on the reconstructed image includes the steps of dividing the reconstructed image to obtain a plurality of regions, and determining, as the at least one region, a region in the plurality of regions whose variance is greater than a first threshold, or determining, for each of the plurality of regions, the ratio of the pixels whose gradient is greater than a second threshold among all the pixels in the region, and determining, as the at least one region, a region whose ratio is greater than a third threshold. For example, if the variance of a region is greater than a threshold t1 and t1 > 0, it may be considered that the texture of the region is complex, and thus, it may be necessary to perform enhancement processing to improve the image quality. Or, the gradient is greater than a second threshold, and the ratio of the pixels among all the pixels in the region is greater than a third threshold. For example, if the gradient is greater than a threshold a, and the ratio of the pixels existing in all the pixels in the region is greater than a threshold t2, where a > 0 and 0 < t2 < 1, it may be considered that the texture of the region is complex, and thus, it is necessary to perform enhancement processing to improve the image quality.

[0035] For the base layer, the encoder encodes the original image to obtain a base layer bitstream, and then the base layer bitstream is decoded to obtain a reconstructed image of the base layer. For example, a VVC encoder encodes the original image to obtain a base layer bitstream, and a VVC decoder decodes the base layer bitstream to obtain a reconstructed image of the base layer. It should be understood that another encoder, an HEVC codec, or an AVC codec may alternatively be used for the base layer. This is not particularly limited in the embodiments of the present application.

[0036] In the above-mentioned multiple regions, based on the attributes or features of the image, not all regions need to be enhanced, and only the encoding and decoding of the base layer need to be performed for one or more regions to meet the image quality requirements. Therefore, in this embodiment of the present application, the target region is the region to be enhanced. In the subsequent encoding of the enhancement layer, the encoding and decoding of the enhancement layer can be performed only for the target region. In this way, the entire image does not need to be enhanced, and the encoding and decoding efficiency of the image can be improved.

[0037] In a possible implementation, the method further comprises using side information of the residual feature map as reconstructed side information of the residual feature map.

[0038] According to a second aspect, the present application provides a method for decoding a regional enhancement layer, the method including: obtaining reconstructed pixels of a base layer for a target region; inputting the reconstructed pixels into a correction network to obtain correction information for the target region; obtaining an enhancement layer bitstream for the target region; decoding the enhancement layer bitstream to obtain a residual feature map of an enhancement layer for the target region; inputting the residual feature map and the correction information into a decoding network to obtain reconstructed pixels of the enhancement layer for the target region.

[0039] In this embodiment of the present application, the correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then decodes the enhancement layer bitstream based on the correction information. The decoding of the enhancement layer is performed only for the necessary region (target region), which can reduce the decoding complexity of the enhancement layer and improve the decoding efficiency of the enhancement layer. In addition, the decoding is performed based on the correction information, which can improve the decoding accuracy.

[0040] In a possible implementation, the step of inputting the reconstructed pixels into a correction network to obtain correction information for the target area includes a step of inputting the reconstructed pixels into the correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​for the target area, wherein the correction information is a plurality of pixel values ​​or a plurality of feature values.

[0041] In a possible implementation, the step of decoding the enhancement layer bitstream to obtain a residual feature map of the enhancement layer for the target region includes the steps of: obtaining a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream; and performing entropy decoding on corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain a residual feature map.

[0042] In a possible implementation, obtaining the plurality of probability distributions based on the correction information includes inputting the correction information into a probability estimation network to obtain the plurality of probability distributions.

[0043] In a possible implementation, obtaining a plurality of probability distributions based on the correction information includes obtaining a plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map.

[0044] In a possible implementation, the step of obtaining the plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map includes, when the correction information is the plurality of feature values, inputting the reconstructed side information into a side information processing network to obtain a first feature map, and inputting the plurality of feature values ​​and the first feature map into a probability estimation network to obtain the plurality of probability distributions.

[0045] In a possible implementation, the step of obtaining a plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map includes, when the correction information is a plurality of pixel values, inputting the plurality of pixel values ​​into a feature estimation network to obtain a second feature map; inputting the reconstructed side information into a side information processing network to obtain a first feature map; and inputting the first feature map and the second feature map into a probability estimation network to obtain the plurality of probability distributions.

[0046] In a possible implementation, the step of decoding the enhancement layer bitstream to obtain an enhancement layer residual feature map for the target region includes the steps of: obtaining a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream; and performing entropy decoding on corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain the residual feature map.

[0047] In a possible implementation, the step of obtaining a plurality of probability distributions based on the reconstructed side information of the residual feature map includes the step of inputting the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0048] In a possible implementation, obtaining a plurality of probability distributions based on reconstructed side information of the residual feature map includes obtaining a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0049] In a possible implementation, the step of obtaining multiple probability distributions based on the reconstructed side information and the reconstructed pixels includes the steps of inputting the reconstructed pixels into a feature estimation network to obtain a third feature map, inputting the reconstructed side information into a side information processing network to obtain a first feature map, and inputting the first feature map and the third feature map into a probability estimation network to obtain multiple probability distributions.

[0050] In a possible implementation, the method further includes inputting the residual feature map into a side information extraction network to obtain side information of the residual feature map, and using the side information as reconstructed side information of the residual feature map.

[0051] In a possible implementation, the method further comprises the steps of obtaining a side information bitstream of the target region and parsing the side information bitstream to obtain the reconstructed side information.

[0052] In a possible implementation, the decoding network includes a first decoding network, and inputting the residual feature map and the correction information into the decoding network to obtain reconstructed pixels of the enhancement layer of the target region includes inputting the residual feature map into the first decoding network to obtain reconstructed residual pixels of the enhancement layer of the target region, and, if the correction information is a plurality of pixel values, summing the reconstructed residual pixels and corresponding pixel values ​​in the correction information to obtain reconstructed pixels.

[0053] In a possible implementation, the decoding network includes a second decoding network, and the step of inputting the residual feature map and the correction information into the decoding network to obtain reconstructed pixels of the enhancement layer of the target region includes the steps of inputting the residual feature map into the second decoding network, and if the correction information is a plurality of feature values, adding an output of any convolutional layer in the second decoding network and a corresponding feature value in the correction information, and inputting the addition result into a network layer after the any convolutional layer in the second decoding network to obtain reconstructed pixels.

[0054] In a possible implementation, the method further includes the steps of obtaining a base layer bitstream of the image to which the target region belongs, parsing the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determining, based on the reconstructed image, at least one region to be enhanced, wherein the target region is one of the at least one region.

[0055] In a possible implementation, the step of determining at least one region to be enhanced based on the reconstructed image includes the steps of dividing the reconstructed image to obtain a plurality of regions, and determining as at least one region a region among the plurality of regions whose variance is greater than a first threshold, or determining a proportion of pixels whose gradient is greater than a second threshold and that are present in each of the plurality of regions relative to the total pixels, and determining as at least one region a region whose proportion is greater than a third threshold.

[0056] According to a third aspect, the present application provides an encoding apparatus, including: an acquisition module configured to acquire reconstructed pixels of a base layer for a target region; a processing module configured to input the reconstructed pixels to a correction network to obtain correction information for the target region, and to input the correction information and original pixels of the target region to an encoding network to obtain residual feature maps of an enhancement layer for the target region; and an encoding module configured to encode the residual feature maps to obtain an enhancement layer bitstream for the target region.

[0057] In a possible implementation, the processing module is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area, where the correction information is a plurality of pixel values ​​or a plurality of feature values.

[0058] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0059] In a possible implementation, the encoding module is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0060] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the correction information of the residual feature map and the reconstructed side information.

[0061] In a possible implementation, the encoding module is specifically configured to input the reconstructed side information into a side information processing network to obtain a first feature map when the correction information is a plurality of feature values, and to input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0062] In a possible implementation, the encoding module is specifically configured to, when the correction information is a plurality of pixel values, input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0063] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0064] In a possible implementation, the encoding module is specifically configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0065] In a possible implementation, the encoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0066] In a possible implementation, the encoding module is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0067] In a possible implementation, the encoding module is further configured to input the residual feature map to a side information extraction network to obtain side information of the residual feature map, and then perform entropy coding on the side information or the quantized side information, and write the encoded side information into a bitstream.

[0068] In a possible implementation, the encoding network includes a first encoding network, and the encoding module is configured to, when the correction information is a plurality of pixel values, perform subtraction on corresponding pixel values ​​in the original pixel and the correction information, and input the subtraction result to the first encoding network to obtain a residual feature map.

[0069] In a possible implementation, the encoding network includes a second encoding network, and the encoding module is configured to input the original pixels to the second encoding network, and when the correction information is a plurality of feature values, perform subtraction between the output of any convolutional layer in the second encoding network and the corresponding feature value in the correction information, and input the subtraction result to a network layer after the any convolutional layer in the second encoding network to obtain a residual feature map.

[0070] In a possible implementation, the encoding module is further configured to encode the image to which the target region belongs to obtain a base layer bitstream of the image, decode the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0071] In a possible implementation, the encoding module is configured to divide the reconstructed image to obtain a plurality of regions, and determine as at least one region a region in which the variance in the plurality of regions is greater than a first threshold, or determine a proportion of pixels in each of the plurality of regions whose gradient is greater than a second threshold and is present in the total pixels, and determine as at least one region a region in which the proportion is greater than a third threshold.

[0072] In a possible implementation, the encoding module is further configured to use side information of the residual feature map as reconstructed side information of the residual feature map.

[0073] According to a fourth aspect, the present application provides a decoding device, including: an acquisition module configured to acquire reconstructed pixels of a base layer for a target region; a processing module configured to input the reconstructed pixels to a correction network to acquire correction information for the target region; and a decoding module configured to decode the enhancement layer bitstream to acquire a residual feature map of an enhancement layer for the target region, where the acquisition module is further configured to acquire the enhancement layer bitstream for the target region, and the processing module is further configured to input the residual feature map and the correction information to a decoding network to acquire reconstructed pixels of the enhancement layer for the target region.

[0074] In a possible implementation, the processing module is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target region, where the correction information is the plurality of pixel values ​​or the plurality of feature values.

[0075] In a possible implementation, the decoding module is specifically configured to obtain, based on the correction information, a plurality of probability distributions, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream; and to perform entropy decoding on the corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain a residual feature map.

[0076] In a possible implementation, the decoding module is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0077] In a possible implementation, the decoding module is specifically configured to obtain the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map.

[0078] In a possible implementation, the decoding module is specifically configured to, when the correction information is a plurality of feature values, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0079] In a possible implementation, the decoding module is specifically configured, when the correction information is a plurality of pixel values, to input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, to input the reconstructed side information into a side information processing network to obtain a first feature map, and to input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0080] In a possible implementation, the decoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and to perform entropy decoding on the corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain the residual feature map.

[0081] In a possible implementation, the decoding module is specifically configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0082] In a possible implementation, the decoding module is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0083] In a possible implementation, the decoding module is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0084] In a possible implementation, the decoding module is further configured to input the residual feature map to a side information extraction network to obtain side information of the residual feature map, and use the side information as reconstructed side information of the residual feature map.

[0085] In a possible implementation, the decoding module is further configured to obtain a side information bitstream of the target region and to parse the side information bitstream to obtain the reconstructed side information.

[0086] In a possible implementation, the decoding network includes a first decoding network, and the decoding module is specifically configured to input the residual feature map to the first decoding network to obtain reconstructed residual pixels of the enhancement layer of the target region, and when the correction information is the plurality of pixel values, sum the reconstructed residual pixels and corresponding pixel values ​​in the correction information to obtain the reconstructed pixels.

[0087] In a possible implementation, the decoding network includes a second decoding network, and the decoding module is specifically configured to input the residual feature map into the second decoding network, and when the correction information is a plurality of feature values, sum up the output of any convolution layer in the second decoding network and the corresponding feature value in the correction information, and input the summation result into a network layer after the any convolution layer in the second decoding network to obtain a reconstructed pixel.

[0088] In a possible implementation, the decoding module is further configured to obtain a base layer bitstream of the image to which the target region belongs, parse the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0089] In a possible implementation, the decoding module is specifically configured to divide the reconstructed image to obtain a plurality of regions, and determine as at least one region a region in which the variance in the plurality of regions is greater than a first threshold, or determine a proportion of pixels in each of the plurality of regions whose gradient is greater than a second threshold and is present in the total pixels, and determine as at least one region a region in which the proportion is greater than a third threshold.

[0090] According to a fifth aspect, the present application provides an encoder including one or more processors and a memory configured to store one or more programs, the one or more programs, when executed by the one or more processors, enabling the one or more processors to perform a method according to any one of the possible implementations of the first aspect.

[0091] According to a sixth aspect, the present application provides a decoder, comprising one or more processors and a memory configured to store one or more programs, the one or more programs, when executed by the one or more processors, enabling the one or more processors to perform a method according to any one of the possible implementations of the second aspect.

[0092] According to a seventh aspect, the present application provides a computer-readable storage medium comprising a computer program which, when executed on a computer, enables the computer to carry out a method according to any one of the possible implementations of the first and second aspects.

[0093] According to an eighth aspect, the present application provides a computer program product, the computer program product comprising instructions which, when executed on a computer or processor, enable the computer or processor to perform a method according to any one of the possible implementation forms of the first and second aspects. [Brief explanation of the drawings]

[0094] [Figure 1] FIG. 1 is an exemplary diagram of layers of scalable video coding according to the present application. [Figure 2A] FIG. 2A is a block diagram of an exemplary coding system 10. [Figure 2B] FIG. 2B is an exemplary diagram of a video coding system 40. [Figure 3] FIG. 3 is a diagram illustrating a video coding device 400 according to one embodiment of the present application. [Figure 4] FIG. 4 is a simplified block diagram of an example of an apparatus 500, according to one embodiment. [Figure 5] FIG. 5 is a diagram of an application scenario according to one embodiment of the present application. [Figure 6] FIG. 6 is a diagram of an application scenario according to one embodiment of the present application. [Figure 7] FIG. 7 is a diagram of an application scenario according to one embodiment of the present application. [Figure 8] FIG. 8 is a flow chart of a method process 800 for encoding a region enhancement layer according to one embodiment of the present application. [Figure 9A] FIG. 9a is an exemplary diagram of a correction network. [Figure 9B] FIG. 9b is an exemplary diagram of a correction network. [Figure 9C] FIG. 9c is an exemplary diagram of a correction network. [Figure 9D] FIG. 9d is an exemplary diagram of a correction network. [Figure 10A] FIG. 10a is an exemplary diagram of an encoding network. [Figure 10B] FIG. 10b is an exemplary diagram of an encoding network. [Figure 11A] FIG. 11a is an exemplary diagram of a probability estimation network. [Figure 11B] FIG. 11b is an exemplary diagram of a probability estimation network. [Figure 12] FIG. 12 is a flow chart of a process 1200 of a method for decoding a region enhancement layer according to one embodiment of the present application. [Figure 13A] FIG. 13a is an exemplary diagram of a decoding network. [Figure 13B] FIG. 13b is an exemplary diagram of a decoding network. [Figure 14]FIG. 14 is an exemplary diagram of the encoding and decoding process. [Figure 15] FIG. 15 is an exemplary diagram of the encoding and decoding process. [Figure 16] FIG. 16 is an exemplary diagram of the encoding and decoding process. [Figure 17] FIG. 17 is an exemplary diagram of the encoding and decoding process. [Figure 18] FIG. 18 is an exemplary diagram of the structure of an encoding device 1800 according to an embodiment of the present application. [Figure 19] FIG. 19 is an exemplary diagram of the structure of a decoding device 1900 according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0095] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the accompanying drawings in the present application. It is clear that the described embodiments are not all of the embodiments of the present application, but only a part thereof. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0096] In the specification, embodiments, claims, and accompanying drawings of this application, terms such as "first" and "second" are intended merely to distinguish and describe, and should not be understood as indicating or implying relative importance or sequence. In addition, the terms "include," "have," and any variations thereof are intended to cover a non-exclusive inclusion, e.g., to include a series of steps or units. For example, a method, system, product, or device is not necessarily limited to the explicitly recited steps or units, but may include other steps or units that are not explicitly recited or that are inherent to such a process, method, product, or device.

[0097] In this application, it should be understood that "at least one (item)" means one or more, and "a plurality of" means two or more. The term "and / or" is used to describe an association relationship between related objects and indicates that three relationships may exist. For example, "A and / or B" can represent three cases: only A is present, only B is present, and both A and B are present, where A and B may be singular or plural. The character " / " generally indicates an "or" relationship between related objects. The phrase "at least one of the following items" or similar phrases refers to any combination of these items, including any combination of single items or multiple items. For example, at least one item (piece) of a, b, or c may indicate a, b, c, "a and b," "a and c," "b and c," or "a and b and c," where a, b, and c may be singular or plural.

[0098] FIELD OF THE INVENTION The embodiments of the present application relate to the application of neural networks. For ease of understanding, the following will first explain and describe relevant nouns or terms.

[0099] 1. Neural Networks

[0100] A neural network (NN) is a machine learning model. A neural network may contain neurons. A neuron takes as input x s and an operation unit with an intercept of 1, where the output of the operation unit may be:

number

[0101] s=1,2,…,n, where n is a natural number greater than 1, and W s x s where is the weight of the neuron, and b is the bias of the neuron. f is the activation function of the neuron, which is configured to introduce a non-linear feature into the neural network to convert the input signal at the neuron into an output signal. The output signal of the activation function may be used as the input of the next convolutional layer. The activation function may be a non-linear function such as ReLU. A neural network is a network formed by connecting many single neurons. Specifically, the output of a neuron may be the input of another neuron. The input of each neuron may be connected to the local receptive field of the previous layer to extract features of the local receptive field. The local receptive field may be a region containing several neurons.

[0102] 2. Multi-layer perceptron (MLP)

[0103] MLP is a simple deep neural network (DNN) (different layers are fully connected), and is also called multi-layer neural network. MLP can be understood as a neural network with multiple hidden layers. There is no special metric for "multiple" in this specification. DNN is divided based on the position of different layers, and neural networks in DNN can be divided into three types: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layer is a hidden layer. The layers are fully connected. Specifically, the i-th layer (i th The neurons in the (i+1)th layer ((i+1) th Although DNNs appear complex, in reality, the work done in each layer is not complicated and can be expressed simply as the following linear relationship:

number

number

number

number

number

number

number

[0104] 3. Convolutional Neural Networks

[0105] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. In a deep learning architecture, multi-layer learning is performed at different abstraction levels according to a machine learning algorithm. As a deep learning architecture, a CNN is a feed-forward artificial neural network. Neurons in a feed-forward artificial neural network can respond to images input into the neural network. A convolutional neural network includes a feature extractor composed of a convolutional layer and a pooling layer. The feature extractor can be viewed as a filter. The convolution process can be viewed as using a trainable filter to perform convolution on an input image or a convolutional feature plane (feature map).

[0106] A convolutional layer is a neuron layer within a convolutional neural network where a convolution operation is performed on an input signal. A convolutional layer may include multiple convolution operators. A convolution operator is also called a kernel. In image processing, a convolution operator functions as a filter that extracts specific information from an input image matrix. A convolution operator may essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix is ​​usually used to process pixels with a granularity of one pixel (or two pixels depending on the stride value) in the horizontal direction of the input image to extract specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix spans the entire depth of the input image. Therefore, a convolution output with a single depth dimension is generated through convolution with a single weight matrix. However, in most cases, a single weight matrix is ​​not used, but multiple weight matrices with the same size (rows by columns), i.e., multiple homogeneous matrices, are applied. The outputs of the weight matrices can be stacked to form the depth dimension of the convolved image. The dimension in this specification can be understood as being determined based on the above-mentioned "multiple." Different weight matrices can be used to extract different features from an image. For example, one weight matrix is ​​used to extract side information of the image, another weight matrix is ​​used to extract a specific color of the image, and yet another weight matrix is ​​used to blur unwanted noise in the image. The sizes (rows by columns) of the multiple weight matrices are the same. The sizes of the feature maps extracted from the multiple weight matrices with the same size are also the same, and then the multiple extracted feature maps with the same size are combined to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications.Each weight matrix containing weight values ​​obtained through training may be used to extract information from an input image, so that the convolutional neural network performs correct prediction. When a convolutional neural network has multiple convolutional layers, a large number of general features are usually extracted in the first convolutional layer. General features are also called low-level features. As the depth of a convolutional neural network increases, the features extracted in subsequent convolutional layers become more complex, such as high-level semantic features. Features with higher-level semantics are more applicable to the problem to be solved.

[0107] The amount of training parameters often needs to be reduced. Therefore, pooling layers often need to be introduced periodically after convolutional layers. One pooling layer may follow one convolutional layer, or multiple convolutional layers may be followed by one or more pooling layers. During image processing, pooling layers are used only to reduce the spatial size of an image. Pooling layers may include average pooling operators and / or max pooling operators to perform sampling on an input image to obtain an image with a small size. The average pooling operator may be used to calculate pixel values ​​in an image within a specific range to generate an average value. The average value is used as the average pooling result. The max pooling operator may be used to select a pixel with the maximum value within a specific range as the max pooling result. In addition, just as the size of the weight matrix in a convolutional layer needs to be related to the size of the image, the operators in a pooling layer also need to be related to the size of the image. The size of the output image after processing in the pooling layer is smaller than the size of the image input to the pooling layer. Each pixel in the image output from the pooling layer represents the average or maximum value of the corresponding sub-area of ​​the image input to the pooling layer.

[0108] After the processing performed in the convolutional / pooling layers, the convolutional neural network is not ready to output the required output information. As described above, in the convolutional / pooling layers, only features are extracted and the resulting parameters are reduced from the input image. However, to generate the final output information (required class information or other related information), the convolutional neural network needs to use neural network layers to generate the output of one required class or a group of required classes. Therefore, the convolutional neural network layers may include multiple hidden layers. The parameters included in the multiple hidden layers may be obtained through pre-training based on relevant training data for a specific task type. For example, task types may include image recognition, image classification, and super-resolution image reconstruction.

[0109] Optionally, in the neural network layer, multiple hidden layers are followed by an output layer of the entire convolutional neural network. The output layer has a loss function similar to categorical cross-entropy, and the loss function is specifically used to calculate the prediction error. Once forward propagation through the entire convolutional neural network is completed, back propagation begins to update the weight values ​​and deviations of each of the above layers to reduce the loss of the convolutional neural network and the error between the results output by the convolutional neural network and the ideal results using the output layer.

[0110] 4. Recurrent Neural Networks

[0111] Recurrent neural networks (RNNs) are used to process sequence data. Traditional neural network models have fully connected layers from the input layer to the hidden layer and output layer, and the nodes in each layer are disconnected. While such general neural networks can solve many problems, they still cannot solve many others. For example, to predict the next word in a sentence, the previous word must often be used because the previous and next words in the sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is also related to the previous output of the sequence. In a specific representation, the network stores previous information and applies it to the calculation of the current output. Specifically, the nodes in the hidden layer are connected, and the input of the hidden layer includes the output of the input layer and also the output of the hidden layer at the previous time point. Theoretically, RNNs can process sequence data of any length. Training an RNN is similar to training a traditional CNN or DNN. The error backpropagation algorithm is also used, but with the following difference: when an RNN is extended, parameters such as W of the RNN are shared. This is different from the traditional neural network described in the example above. Also, while using the gradient descent algorithm, the output at each step depends on the network state at the current step, and also on the network state at several previous steps. The learning algorithm is called the backpropagation through time (BPTT) algorithm.

[0112] Why are recurrent neural networks still needed when convolutional neural networks are available? The reason is simple. Convolutional neural networks assume that elements are independent of each other, like cats and dogs, and that inputs and outputs are also independent. However, in the real world, multiple elements are interconnected. For example, stocks change over time. Another example is when someone says, "I love traveling, and my favorite place is Yunnan. If I have the chance, I'll go there." Here, people should know that the person is going to "Yunnan" because they can make inferences from context. But how does a machine do this? Then, RNNs come into play. RNNs are intended to enable machines to memorize, just like humans. Therefore, the output of an RNN must depend on current input information and historical memory information.

[0113] 5. Loss Function

[0114] In the process of training a deep neural network, the network's current predicted value can be compared with the actual expected target value, since the output of the deep neural network is expected to be as close as possible to the actual expected predicted value. Then, the weight vector of each layer of the neural network is updated based on the difference between the predicted value and the target value (indeed, before the first update, there is usually an initialization process, specifically, parameters are preconfigured for all layers of the deep neural network). For example, if the network's predicted value is large, the weight vector is adjusted to reduce the predicted value, and continuous adjustments are performed until the deep neural network can predict the actual expected target value or a value close to the actual expected target value. Therefore, "how to obtain the difference between the predicted value and the target value through comparison" needs to be defined in advance. This is the loss function or objective function. The loss function and objective function are important equations that measure the difference between the predicted value and the target value. The loss function is used as an example. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training a deep neural network is a process of minimizing the loss as much as possible.

[0115] 6. Backpropagation Algorithm

[0116] A convolutional neural network can correct the parameter values ​​of the initial super-resolution model during the training process using an error backpropagation (BP) algorithm. As a result, the error loss associated with the reconstruction of the super-resolution model is smaller. Specifically, the input signal is forward-transferred until an error loss occurs at the output, and the parameters of the initial super-resolution model are updated based on the error backpropagation information so that the error loss converges. The backpropagation algorithm is an error-loss-centric backpropagation motion designed to obtain parameters, such as a weight matrix, of the optimal super-resolution model.

[0117] 7. Generative Adversarial Networks

[0118] A generative adversarial network (GAN) is a deep learning model. It includes at least two modules: one is a generative model and the other is a discriminative model. The two modules learn from each other through game learning to generate better outputs. Both the generative model and the discriminative model may be neural networks, specifically, deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: A GAN for generating pictures is used as an example, and it is assumed that there are two networks: G (generator) and D (discriminator). G is a network for generating pictures. G receives random noise z and generates a picture by using the noise. Here, the picture is denoted as G(z). D is a discriminator network used to determine whether the picture is "real" or not. The input parameter of D is x, where x represents a picture, and the output D(x) represents the probability that x is an actual picture. A value of 1 for D(x) indicates that the picture is 130% realistic. A value of 0 for D(x) indicates that the picture cannot be realistic. In the process of training a generative adversarial network, the goal of the generative network G is to generate pictures that are as realistic as possible to deceive the discriminative network D, and the goal of the discriminative network D is to distinguish between pictures generated by G and realistic pictures as well as possible. In this way, a dynamic “gaming” process, specifically an “adversary” in a “generative adversarial network,” exists between G and D.The final game result is that, in an ideal situation, G can generate an image G(z) that is difficult to distinguish from a real image, and D makes it difficult to determine whether the image generated by G is real or not. Specifically, D(G(z))=0.5. In this way, an excellent generative model G is obtained and can be used to generate pictures.

[0119] FIELD OF THE INVENTION For ease of understanding, the following will first explain and describe relevant nouns or terms.

[0120] Layered video coding, also known as scalable video encoding and decoding, is an extension of current video coding standards (commonly known as scalable video coding (SVC), an extension of advanced video coding (AVC) (H.264), or scalable high efficiency video coding (SHVC), an extension of high efficiency video coding (HEVC) (H.265). Scalable video coding is primarily used to solve the problems of packet loss and delay jitter caused by real-time network bandwidth changes in real-time video transmission.

[0121] A basic structure in scalable video coding may be referred to as a layer. In scalable video coding techniques, space domain layering (resolution layering) is performed on original image blocks, resulting in bitstreams of different resolution layers. Resolution is the size of an image block in pixels. The resolution of a lower layer is lower, and the resolution of a higher layer is equal to or higher than the resolution of the lower layer. Alternatively, time domain layering (frame rate layering) is performed on original image blocks, resulting in bitstreams of different frame rate grades. The frame rate may be the amount of image frames included in a video per unit time. The frame rate of a lower layer is lower, and the frame rate of a higher layer is equal to or higher than the frame rate of the lower layer. Alternatively, quality domain layering is performed on original image blocks, resulting in bitstreams of different encoding quality grades. The encoding quality may be video quality. The degree of image distortion in the lower layer is high, and the degree of image distortion in the higher layer is not greater than the degree of image distortion in the lower layer.

[0122] A layer called a base layer is usually the bottom layer in scalable video coding. In spatial domain layering, base layer image blocks are encoded at the lowest resolution. In temporal domain layering, base layer image blocks are encoded at the lowest frame rate. In quality domain layering, base layer image blocks are encoded at the highest QP or lowest bitrate. That is, the base layer is the lowest quality layer in scalable video coding. A layer called an enhancement layer is the layer above the base layer in scalable video coding and may be divided into multiple enhancement layers from bottom to top. The lowest enhancement layer encodes the encoding information obtained by the base layer to obtain a combined bitstream and has a higher encoding resolution, frame rate, or bitrate than the base layer. A higher enhancement layer may encode the encoding information of a lower enhancement layer to obtain a higher quality image block.

[0123] For example, FIG. 1 is an exemplary diagram of layers in scalable video coding according to the present application. As shown in FIG. 1, after an original image block is transmitted to a scalable encoder, the original image block may be layered into a base layer image block B and an enhancement layer image block (E1 to En, n≧1) according to different encoding configurations. Then, a bitstream including a base layer bitstream and an enhancement layer bitstream is obtained through encoding. The base layer bitstream is generally a bitstream obtained from an image block with a minimum resolution, a minimum frame rate, or a minimum encoding quality parameter. The enhancement layer bitstream is a bitstream obtained by jointly encoding image blocks based on the base layer with a high resolution, a high frame rate, or a high encoding quality parameter. As the amount of enhancement layer increases, the number of spatial domain layers, temporal domain layers, or quality layers for encoding also increases. When the encoder transmits a bitstream to a decoder, the transmission of the base layer bitstream is reserved first. When the network has a margin, the bitstreams at the higher layers are transmitted gradually. The decoder first receives and decodes the base layer bitstream, and then decodes the received enhancement layer bitstream from the lower layer to the higher layer, layer by layer, in the spatial domain, the temporal domain, or in ascending order of quality. Then, the decoded information of the higher layer is superimposed on the reconstructed block of the lower layer to obtain a reconstructed block with a higher resolution, a higher frame rate, or a higher quality.

[0124] As mentioned above, each image in a video sequence is usually partitioned into a set of non-overlapping blocks, and encoding is usually performed at the block level. In other words, an encoder usually processes video at the block (image block) level, i.e., encodes video, for example, generating a predictive block through spatial (intra-frame) prediction and temporal (inter-frame) prediction, subtracting the predictive block from the image block (the block currently being processed / to be processed) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compressed). The encoder further needs to perform inverse quantization and inverse transform to obtain a reconstructed residual block, and then add pixel values ​​of the reconstructed residual block to pixel values ​​of the predictive block to obtain a reconstructed block. The reconstructed block in the base layer is a reconstructed block obtained by performing the above-mentioned operations on a base layer image block obtained by layering the original image block.

[0125] The region in the following can be the largest coding unit (LCU) in the entire image, an image block obtained by dividing the entire image, a region of interest (ROI) in the entire image (i.e., a specified image region in the image to be processed), or a slice in the image.

[0126] Taking advantage of the excellent performance of deep learning in various fields, researchers have proposed a deep learning-based end-to-end image encoding solution. In combination with this solution, related technology provides a layered image encoding framework, where the base layer uses a codec based on versatile video coding (VVC), and the enhancement layer uses a codec based on artificial intelligence (AI). This solution is described as follows:

[0127] Encoder side:

[0128] For the base layer, a VVC encoder encodes an input image x to obtain a bitstream, and a VVC decoder decodes the bitstream to obtain a reconstructed image xc of the base layer.

[0129] For the enhancement layer, x and xc are input to an encoding network (encoder) to obtain the residual feature map (y) of the enhancement layer. On the branch, y is input to a side information extraction network to obtain the side information (z) of y, and z is the quantized side information (

number

number

number

number

[0130] Decoder side:

[0131] For the base layer, the VVC decoder parses the bitstream to obtain the base layer reconstructed image xc.

[0132] For the enhancement layer, the bitstream contains the decoded side information (z ^ ) and reconstructed side information (z ~ ) to obtain z ^ Inverse quantization is performed on z ~ is y ^The decoded residual feature map y is fed into a probability estimation network to obtain a probability distribution of ^ Entropy decoding is performed on the bitstream based on the probability distribution to obtain the reconstructed residual feature map (y ~ ) to obtain y ^ Inverse quantization is performed on xc and y ~ is input to the decoding network (Decoder) to obtain the reconstructed image yc of the enhancement layer.

[0133] In the above solution, AI image encoding and decoding methods are used in the enhancement layer, and are better than traditional image encoding solutions in terms of objective quality measures such as MS structural similarity (SSIM) and peak signal-to-noise ratio (PSNR). However, in the above solution, the AI ​​image encoding and decoding has high network complexity and high computational requirements. As a result, the encoding and decoding efficiency of the enhancement layer is low.

[0134] Therefore, one embodiment of the present application provides a method for encoding and decoding a region enhancement layer to improve the encoding and decoding efficiency of the enhancement layer. The following describes systems and / or scenarios to which the solution in the embodiment of the present application can be applied.

[0135] 2A is a block diagram of an exemplary coding system 10. A video encoder 20 (or simply referred to as encoder 20) and a video decoder 30 (or simply referred to as decoder 30) in the coding system 10 may be configured to implement various exemplary solutions described in the embodiments of the present application.

[0136] 2A, coding system 10 includes a source device 12. Source device 12 is configured to provide encoded image data 21, e.g., an encoded image, to a destination device 14 for decoding the encoded image data 21.

[0137] The source device 12 includes an encoder 20 and may additionally, i.e., optionally, include an image source 16, a preprocessor (or preprocessing unit) 18, e.g., an image preprocessor, and a communication interface (or communication unit) 22.

[0138] Image source 16 may include or be any kind of image capture device, e.g., a camera for capturing real-world images, and / or any type of image generation device, e.g., a computer graphics processing unit for generating computer-animated images, or any type of other device for obtaining and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory or storage device that stores any of the above-mentioned images.

[0139] To distinguish the processing performed by the preprocessor (or preprocessing unit 18 ), the image (or image data) 17 is also referred to as the original image (or original image data) 17 .

[0140] The pre-processor 18 is configured to receive (original) image data 17 and perform pre-processing on the image data 17 to obtain a pre-processed image (or pre-processed image data) 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component.

[0141] Video encoder (or encoder) 20 is configured to receive pre-processed image data 19 and provide encoded image data 21 (further details are described below, e.g., with reference to FIG. 3).

[0142] The communication interface 22 of the source device 12 may be configured to receive the encoded image data 21 and transmit the encoded image data 21 (or any further processed version thereof) via the communication channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0143] Destination device 14 includes a decoder 30 and may additionally, i.e., optionally, include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32, and a display device 34.

[0144] The communications interface 28 of the destination device 14 is configured to receive the encoded image data 21 (or any further processed version thereof), for example, directly from the source device 12 or from any other source device, for example, a storage device, for example, an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0145] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded image data (or encoded data) 21 via a direct communication link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network, or any combination thereof, or any type of private and public network, or any combination thereof.

[0146] The communications interface 22 may be configured, for example, to package the encoded image data 21 into an appropriate format, e.g., packets, and / or process the encoded image data using any type of transmission encoding or processing for transmission over a communications link or network.

[0147] Communications interface 28, which forms a counterpart to communications interface 22, may be configured, for example, to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or depackaging to obtain encoded image data 21.

[0148] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in Figure 2A, or as bidirectional communication interfaces, and may be configured, for example, to send and receive messages, for example, to set up connections, acknowledge and exchange any other information related to the communication link and / or data transmission, e.g., encoded image data transmission.

[0149] Video decoder (or decoder) 30 is configured to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (further details are described below, e.g., based on FIG. 4).

[0150] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), e.g., a decoded image, to obtain post-processed image data 33, e.g., a post-processed image. The post-processing performed by the post-processor 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, resampling, or any other processing to prepare the decoded image data 31 for display by, for example, a display device 34.

[0151] The display device 34 is configured to receive the post-processed image data 33, for example, to display the image to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0152] Coding system 10 further includes a training engine 25. Training engine 25 is configured to train encoder 20 or decoder 30, and in particular, the neural network (described in detail below) used in encoder 20 or decoder 30.

[0153] In an embodiment of the present invention, the training data may be stored in a database (not shown), and the training engine 25 performs training based on the training data to obtain a neural network. It should be noted that the source of the training data is not limited in the embodiment of the present application. For example, the training data may be obtained from the cloud or other locations to perform model training.

[0154] 2A illustrates source device 12 and destination device 14 as separate devices, device embodiments may also include both source device 12 and destination device 14, or the functionality of both source device 12 and destination device 14, i.e., both source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, by separate hardware and / or software, or by any combination thereof.

[0155] Following this description, it will be clear to those skilled in the art that the presence and (exact) division of different units or functions in the source device 12 and / or destination device 14 shown in FIG. 2A may vary depending on the actual device and application.

[0156] The encoder 20 (e.g., video encoder 20), or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented via processing circuitry such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated processors for video coding, or any combination thereof, as shown in FIG. 2B . The encoder 20 and the decoder 30 may be implemented separately using processing circuitry 46. The processing circuitry 46 may be configured to perform various operations, as described below. When the techniques are implemented partially in software, the device may store instructions for the software on a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of the present application. Either the encoder 20 and the decoder 30 may be integrated as part of a combined encoder / decoder (encoder / decoder, CODEC) in a single device, for example as shown in FIG. 2B.

[0157] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), etc., and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may include components for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0158] In some cases, the coding system 10 shown in FIG. 2A is merely an example, and the techniques of the present application may be applicable to video coding devices (e.g., video encoding or video decoding) that do not necessarily include data communication between the encoding device and the decoding device. In other examples, data may be retrieved from local memory, transmitted over a network, etc. A video encoding device may encode data and store the encoded data in memory. And / or a video decoding device may retrieve data from memory and decode the data. In some cases, encoding and decoding are performed by devices that do not communicate with each other but simply encode data to memory and / or retrieve data from memory and decode the data.

[0159] 2B is an example diagram of a video coding system 40. Video coding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0160] 2B, imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 may be in communication with each other. Video coding system 40 may include only video encoder 20 or only video decoder 30 in different embodiments.

[0161] In some embodiments, antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Additionally, in some embodiments, display device 45 may be configured to present the video data. Processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, etc. Video coding system 40 may also include optional processor 43. Optional processor 43 may also include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, etc. Additionally, memory 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory 44 may be implemented by a cache memory. In another example, processing circuitry 46 may include memory (e.g., a cache) for implementing an image buffer.

[0162] In some examples, video encoder 20 implemented using logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented using processing circuitry 46. The logic circuitry may be configured to perform various operations described herein.

[0163] In some examples, video decoder 30 may be implemented by processing circuitry 46 in a similar manner to implement various modules described with reference to video decoder 30 in FIG. 4 and / or any other decoder system or subsystem described herein. In some embodiments, video decoder 30 implemented using logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 implemented using processing circuitry 46.

[0164] In some embodiments, antenna 42 may be configured to receive an encoded bitstream of video data. As described, the encoded bitstream may include data related to the video frame encoding described herein, indicators, index values, mode selection data, etc., such as data related to encoding partitioning (e.g., transform coefficients or quantized transform coefficients, optional indicators (as described), and / or data defining the encoding partitioning). Video coding system 40 may further include a video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0165] In embodiments of the present application, for examples described with reference to video encoder 20, it should be understood that video decoder 30 may be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 may be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some embodiments, video encoder 20 may entropy encode the syntax elements into an encoded video bitstream. In such examples, video decoder 30 may parse such syntax elements and decode the associated video data accordingly.

[0166] For ease of explanation, embodiments of the present invention are described herein with reference to reference software for, for example, High-Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC), the next-generation video coding standard developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Joint Collaboration Team on Video Coding (JCT-VC) of the Motion Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0167] 3 is a diagram of a video coding device 400 according to one embodiment of the present invention. The video coding device 400 is suitable for implementing the embodiments described and disclosed herein. In one embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 in FIG. 2A, or an encoder, such as the video encoder 20 in FIG. 2A.

[0168] Video coding device 400 includes ingress ports 410 (or input ports 410) and receiver units (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 (e.g., processor 430 may be a neural network processing unit 430) for processing the data, transmitter units (Tx) 440 and egress ports 450 (or output ports 450) for transmitting the data, and memory 460 for storing the data. Video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to ingress ports 410, receiver units 420, transmitter units 440, and egress ports 450 for the emission or ingress of optical or electrical signals.

[0169] The processor 430 is implemented in hardware and software. The processor 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network (NN)-based coding module 470). The coding module 470 implements the above-disclosed embodiments. For example, the coding module 470 performs, processes, prepares, or provides various encoding operations. Thus, the coding module 470 provides substantial improvements to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0170] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device, storing programs when such programs are selected for execution, and storing instructions and data read during program execution. Memory 460 may be volatile and / or nonvolatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0171] 4 is a simplified block diagram of an apparatus 500 according to one embodiment, which may be used as either or both of source device 12 and destination device 14 in FIG. 2A.

[0172] Processor 502 in apparatus 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or devices now existing or later developed that can manipulate or process information. While the disclosed implementations may be implemented with a single processor, e.g., processor 502, as shown, advantages in speed and efficiency may be achieved using two or more processors.

[0173] In one implementation, the memory 504 in the device 500 may be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable class of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 via a bus 512. The memory 504 may further include an operating system 508 and application programs 510. The application programs 510 include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N and further include a video coding application that performs the methods described herein.

[0174] The apparatus 500 may include one or more output devices, such as a display 518. In one embodiment, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0175] Although bus 512 of device 500 is described herein as a single bus, bus 512 may include multiple buses. Additionally, secondary storage may be directly coupled to other components of device 500 or may be accessed over a network, and may include multiple units, such as a single integrated unit, e.g., a memory card, or multiple memory cards. Accordingly, device 500 may have a wide variety of configurations.

[0176] 5 is a diagram of an application scenario according to one embodiment of the present application. As shown in FIG. 5, the application scenario may be a service related to image / video capture, storage, or transmission in a terminal, a cloud server, or video surveillance. For example, the application scenario may be a photo / video recording in a terminal, an album, a cloud album, or video surveillance.

[0177] Encoder side: A camera captures an image / video. An artificial intelligence (AI) image / video coding network obtains low-redundancy image features from feature extraction for the image / video, and then performs compression based on the image features to obtain a bitstream / image file.

[0178] Decoder side: When an image / video needs to be output, the AI ​​image / video decoding network decompresses the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain a reconstructed image / video.

[0179] The storage / transmission module stores (e.g., captured by a terminal, video surveillance, or cloud server) or transmits (e.g., cloud services or live broadcast technology) the bitstreams / image files obtained by compression for different services.

[0180] 6 is a diagram of an application scenario according to one embodiment of the present application. As shown in FIG. 6, the application scenario may be a service related to image / video capture, storage, or transmission in a terminal or video surveillance, such as a terminal album, video surveillance, or live broadcast.

[0181] Encoder side: The encoding network transforms the image / video into image features with lower redundancy. The encoding network usually includes a nonlinear transformation unit and has nonlinear features. The entropy estimation network is responsible for calculating the encoding probability for each piece of data in the image features. The entropy coding network performs lossless encoding on the image features based on the probability corresponding to each piece of data to obtain a bitstream / image file. This further reduces the amount of data transmission in the image compression process.

[0182] Decoder side: The entropy decoding network performs lossless decoding on the bitstream / image file based on the probability corresponding to each piece of data to obtain reconstructed image features. The decoding network performs inverse transformation on the image features output through entropy decoding and parses the image features into an image / video. Corresponding to the encoding network, the decoding network usually includes a nonlinear transformation unit and has nonlinear features. The storage module saves the bitstream / image file to a corresponding storage location of the terminal. The loading module loads the bitstream / image file from the corresponding storage location of the terminal and inputs the bitstream / image file into the entropy decoding network.

[0183] 7 is a diagram of an application scenario according to one embodiment of the present application. As shown in FIG. 7, the application scenario may be a service related to image / video capture, storage, or transmission in the cloud or video surveillance, such as cloud album, video surveillance, or live broadcast.

[0184] Encoder side: The local side acquires images / videos, encodes the images / videos (JPEG) to obtain compressed images / videos, and then sends the compressed images / videos to the cloud. The cloud performs JPEG decoding on the compressed images / videos to obtain images / videos, and then compresses the images / videos to obtain bitstreams / image files, and stores the bitstreams / image files.

[0185] Decoder side: When the local side needs to obtain images / videos from the cloud, the cloud decompresses the bitstream / image file to obtain the images / videos, performs JPEG encoding on the images / videos to obtain compressed images / videos, and then sends the compressed images / videos to the local side. The local side performs JPEG decoding on the compressed images / videos to obtain the images / videos. For the structure of the cloud and the usage of each module, please refer to the structure and usage of each module in Figure 7. Details will not be described in this specification in the embodiments of the present application.

[0186] Based on the above encoding / decoding network and application scenarios, the embodiments of the present application provide an image encoding / decoding method to implement efficient nonlinear transform processing and improve the rate-distortion performance of image / video compression algorithms.

[0187] 8 is a flowchart of a process 800 of a method for encoding a region enhancement layer according to one embodiment of the present application. The process 800 may be performed by the video encoder 20. The process 800 is described as a series of steps or operations. It should be understood that the steps or operations of the process 800 may be performed in various orders and / or simultaneously and are not limited to the order of execution shown in FIG. 8. Assuming a video data stream having multiple image frames is used by the video encoder 20, the process 800 is performed to encode the region enhancement layer, including the following steps:

[0188] Step 801: Obtain the reconstructed pixels of the base layer of the target region.

[0189] Typically, each image in a video sequence may be partitioned into a set of non-overlapping image blocks. Then, encoding is performed at the image block level. In other words, an encoder usually processes video at the block (image block) level, i.e., encodes video, for example, by generating a predictive block through spatial (intra-frame) prediction and temporal (inter-frame) prediction, subtracting the predictive block from the image block (the block currently being processed / to be processed) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compressed). The encoder further needs to perform inverse quantization and inverse transform to obtain a reconstructed residual block, and then add pixel values ​​of the reconstructed residual block to pixel values ​​of the predictive block to obtain a reconstructed block. In this embodiment of the present application, the region included in the image frame may be a largest coding unit (LCU) in the entire image, an image block obtained by dividing the entire image, a region of interest (ROI) in the entire image (i.e., a designated image region to be processed in the image), etc. In addition to the several cases described above, it should be understood that a region may alternatively be a sub-image described in another manner, which is not specifically limited herein.

[0190] Based on this, the target area is intended to represent the location of an image block that is focused and processed in one encoding process in the solution of this embodiment of the present application. The shape of the target area may be a regular rectangle or square, or may be an irregular shape, which is not particularly limited in this specification.

[0191] As mentioned above, in the encoding process, an initially obtained image block may be referred to as an original block, and a pixel included in the image block may be referred to as an original pixel. A reconstructed image block may be referred to as a reconstructed block, and a pixel included in the reconstructed image block may be referred to as a reconstructed pixel. However, in scalable video coding, although different layers have different resolutions, frame rates, or qualities, the encoding process is generally similar, and in particular, each layer includes an initial image block and a reconstructed image block.

[0192] Therefore, in this embodiment of the present application, there may be the following concepts for the target area:

[0193] In the base layer, the pixels contained in the originally obtained region are referred to as the original pixels of the region's base layer, and the pixels contained in the reconstructed region are referred to as the reconstructed pixels of the region's base layer.

[0194] In the enhancement layer, the pixels contained in the originally acquired region are referred to as the original pixels of the enhancement layer of the region, and the pixels contained in the reconstructed region are referred to as the reconstructed pixels of the enhancement layer of the region.

[0195] It should be noted that the concept of pixel quantity is not mentioned in the above description of original pixels or reconstructed pixels, but it should be understood that a region often includes multiple pixels, which is not particularly limited in the embodiments of the present application.

[0196] In this embodiment of the present application, obtaining reconstructed pixels of the base layer of the target region may include: encoding an image to which the target region belongs to obtain a base layer bitstream of the image; decoding the base layer bitstream to obtain a reconstructed image of the base layer of the image; and determining at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0197] For the base layer, the encoder encodes the original image to obtain a base layer bitstream, and then the base layer bitstream is decoded to obtain a reconstructed image of the base layer. For example, a VVC encoder encodes the original image to obtain a base layer bitstream, and a VVC decoder decodes the base layer bitstream to obtain a reconstructed image of the base layer. It should be understood that another encoder, an HEVC codec, or an AVC codec can alternatively be used for the base layer. This is not particularly limited in the embodiments of the present application.

[0198] After the reconstructed image of the base layer is obtained, the reconstructed image may be divided to obtain multiple regions. For multiple regions, see the related descriptions of the above regions, such as multiple LCUs, multiple image blocks, and multiple ROIs. It should be noted that a region may be a partial image described in multiple ways. Correspondingly, multiple division methods may be used to obtain multiple regions. This is not particularly limited in the embodiment of the present application.

[0199] In the above-mentioned multiple regions, based on the attributes or features of the image, it is not necessary for all regions to be enhanced, and only the encoding and decoding of the base layer need to be performed on one or more regions to meet the image quality requirements. Therefore, in this embodiment of the present application, the target region is the region to be enhanced. In the subsequent encoding of the enhancement layer, the encoding and decoding of the enhancement layer can be performed only on the target region. In this way, it is not necessary to emphasize the entire image, and the encoding and decoding efficiency of the image can be improved.

[0200] At least one of the multiple regions can be determined as the target region. At least one region satisfies the following conditions. That is, the variance of the region is greater than the first threshold. For example, when the variance of the region is greater than the threshold t1 and t1 > 0, the texture of the region may be considered complex, and therefore, enhancement processing needs to be performed to improve the image quality. Or the gradient is greater than the second threshold, and the pixels of the pixels among all the pixels in the region are greater than the third threshold. For example, when the gradient is greater than the threshold a, and the pixels of all the pixels in the region are greater than the threshold t2, a > 0, and 0 < t2 < 1, the texture of the region may also be considered complex, and therefore, enhancement processing needs to be performed to improve the image quality.

[0201] After the above-mentioned process, the encoder side uses any one of at least one region as the current target region, and can extract the pixels at the position corresponding to the target region from the reconstructed image of the base layer, that is, the reconstructed pixels of the base layer of the target region

[0202] Step 802: Input the reconstructed pixels into the correction network to obtain the correction information of the target region.

[0203] The input to the correction network is the reconstructed pixels of the base layer of the target region, and the output is correction information corresponding to the target region. The correction network functions to remove noise signals that are not useful for encoding the AI ​​enhancement layer. The correction network can be formed by a convolutional layer (conv) and an activation layer (ReLU). Whether the correction network includes an activation layer (generalized decomposition normalization, GDN) is not limited, and whether a separate activation function exists is also not limited. In this embodiment of the present application, the number of convolutional layers is not limited, and the size of the convolutional kernel is also not limited. For example, the convolutional kernel can be 3x3, 5x5, or 7x7.

[0204] In a possible implementation, the reconstructed pixels of the base layer of the target region are input to a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target region, where the correction information may be the plurality of pixel values ​​or the plurality of feature values.

[0205] In this embodiment of the present application, the correction network may be implemented using a neural network. For example, the correction network may be constructed using a neural network formed by interpolating and cascading four convolutional / deconvolutional layers and three activation layers. The size of the convolutional kernel of each convolutional layer may be set to 3x3, the number of channels for outputting the feature map may be set to M, and downsampling by a factor of 2 is performed on the width and height of each convolutional layer. It should be understood that the above example does not constitute a specific limitation. The size of the convolutional kernel, the number of channels of the feature map, the downsampling coefficient, the number of downsamplings, the number of convolutional layers, and the number of activation layers may all be adjusted. This is not particularly limited in this embodiment of the present application.

[0206] After processing by the correction network, multiple pixel values ​​and / or multiple feature values ​​may be output. It should be understood that the input to the correction network is a reconstructed pixel, and the reconstructed pixel is within the target region. Therefore, even if the output pixel values ​​and / or feature values ​​do not have a one-to-one correspondence with the pixels in the target region, the multiple pixel values ​​and / or feature values ​​may still be considered to be within the target region, i.e., the multiple pixel values ​​and / or feature values ​​may correspond to the target region. For example, the correction information may be an upsampled value of the reconstructed pixel of the base layer of the target region. The resolution of the reconstructed pixel of the base layer of the target region may be the same as or different from the resolution of the reconstructed pixel of the enhancement layer of the target region. This is not particularly limited in the embodiments of the present application.

[0207] FIG. 9A is an exemplary diagram of a correction network. As shown in FIG. 9A, the correction network includes six convolution layers and five activation layers (ReLU), and the size of the convolution kernel is 3×3. The reconstructed pixels of the base layer of the target region are input to the correction network, and multiple pixel values ​​are output. The correction network can perform denoising on the reconstructed pixels to remove noise signals that are not useful for encoding the AI ​​enhancement layer, and obtain multiple pixel values.

[0208] 9b is an exemplary diagram of the correction network. As shown in FIG. 9b, the correction network includes six convolution layers and five activation layers (ReLU), and the size of the convolution kernel is 3 × 3. The reconstructed pixels of the base layer of the target region are input to the correction network, and multiple feature values ​​are output.

[0209] 9c is an exemplary diagram of a correction network. As shown in FIG. 9c, the correction network includes eight convolution layers and six activation layers (ReLU), and the size of the convolution kernel is 3×3. The reconstructed pixels of the base layer of the target region are input to the correction network, and multiple pixel values ​​and multiple feature values ​​are output from different layers of the correction network.

[0210] 9d is an exemplary diagram of a correction network. As shown in FIG. 9d, the correction network includes five convolution layers, one deconvolution layer, and five ReLU activation layers, and the size of the convolution kernel is 3×3. The size of the convolution kernel of the deconvolution layer of the correction network may be set to 3×3, the number of channels for outputting feature maps may be set to 48 (or may be another value not limited herein), and upsampling by a factor of 2 is performed on the width and height of the deconvolution layer. The number of output channels of the last convolution layer is M. When M is 3, multiple pixel values ​​are output. When M is 48, multiple feature values ​​are output. In this case, the resolution of the correction information is twice the resolution of the reconstructed pixels of the base layer.

[0211] It should be noted that in addition to the above four examples of the correction network, other structures of the correction network may alternatively be used in the embodiments of the present application, which are not particularly limited herein.

[0212] Step 803: Input the correction information and original pixels of the target region into the encoding network to obtain the residual feature map of the enhancement layer of the target region.

[0213] The encoder side can extract pixels at positions corresponding to the target region from the original image, i.e., the original pixels of the target region.

[0214] As mentioned above, the correction information can have two cases: multiple pixel values ​​and multiple features. Correspondingly, the encoding network can also use two structures. In this embodiment of the present application, the input to the encoding network (Encoder) on the encoder side is the correction information and the original pixels of the target region, and the output is the residual feature map of the enhancement layer of the target region.

[0215] 10a is an exemplary diagram of an encoding network. As shown in FIG. 10a, the encoding network may be a first encoding network including four convolution layers and three GDNs. Subtraction is performed on the original pixels and corresponding pixel values ​​in the correction information (multiple pixel values), and then the subtraction result is input to the first encoding network to obtain a residual feature map. In this embodiment, since the correction information (multiple pixel values) corresponds to a pixel region, subtraction may be performed directly on the correction information and the original pixels of the target region.

[0216] FIG. 10b is an exemplary diagram of an encoding network. As shown in FIG. 10b, the encoding network may be a second encoding network including four convolutional layers and three GDNs. An original pixel is input to the second encoding network, and subtraction is performed on the output of any convolutional layer (e.g., the second convolutional layer) in the second encoding network and the corresponding feature values ​​in the correction information (multiple feature values). The subtraction result is then input to a network layer after any convolutional layer (e.g., the second convolutional layer) in the second encoding network to obtain a residual feature map. In this embodiment, since the correction information (multiple feature values) corresponds to a feature region, the original pixel needs to first be input to the second encoding network and converted into the feature region. Then, subtraction is performed on the feature region and the multiple feature values.

[0217] It should be noted that in addition to the above two examples of encoding networks, encoding networks of other structures may alternatively be used in embodiments of the present application, which are not particularly limited herein.

[0218] Step 804: Encode the residual feature map to obtain an enhancement layer bitstream for the target region.

[0219] In this embodiment of the present application, multiple probability distributions may be obtained, where the multiple probability distributions correspond to multiple feature values ​​included in the residual feature map, and then entropy coding is performed on the corresponding feature values ​​in the residual feature map based on the multiple probability distributions respectively to obtain an enhancement layer bitstream.

[0220] The residual feature map of the enhancement layer for the target region contains multiple feature values. To perform entropy coding on the multiple feature values, the probability distributions of the multiple feature values ​​need to be obtained. Below are some methods for obtaining the multiple probability distributions:

[0221] 1. Obtain multiple probability distributions based on correction information

[0222] 1.1. Input the correction information into a probability estimation network to obtain multiple probability distributions.

[0223] The probability estimation network may also include a convolutional layer and a GDN. Whether the probability estimation network has a different activation function is not limited. In this embodiment of the present application, the number of convolutional layers is not limited, and the size of the convolution kernel is not limited. In this embodiment of the present application, a probability distribution model is first used for modeling, and then correction information is input into the probability estimation network to obtain model parameters. The model parameters are then substituted into the probability distribution model to obtain a probability distribution. The probability distribution model may be a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model. When the probability distribution is a Gaussian distribution, the model parameters are the values ​​of the mean parameter μ and variance σ of the Gaussian distribution. When the probability distribution is a Laplacian distribution, the model parameters are the values ​​of the location parameter μ and scale parameter b of the Laplacian distribution. It should be understood that in addition to the above-mentioned probability distribution models, other models may alternatively be used. This is not particularly limited in this specification.

[0224] 11a is an exemplary diagram of a probability estimation network. As shown in FIG. 11a, the probability estimation network includes four convolution layers and three GDNs, and the correction information input to the probability estimation network is a plurality of pixel values.

[0225] 11b is an exemplary diagram of a probability estimation network. As shown in FIG. 11b, the probability estimation network includes two convolutional layers and two GDNs, and the correction information input to the probability estimation network is a plurality of feature values.

[0226] It should be noted that in addition to the above two examples of probability estimation networks, probability estimation networks of other structures may alternatively be used in embodiments of the present application, which are not particularly limited herein.

[0227] 1.2. Obtain multiple probability distributions based on the correction information of the residual feature map and the reconstructed side information

[0228] a. If the correction information is multiple feature values, input the reconstructed side information into a side information processing network to obtain a first feature map, and then input the multiple feature values ​​and the first feature map into a probability estimation network to obtain multiple probability distributions.

[0229] In this embodiment of the present application, the residual feature map of the enhancement layer of the target region may be input to a side information extraction network to obtain side information of the residual feature map, entropy coding is performed on the side information, and the encoded side information is written into a bitstream. In this case, the side information of the residual feature map is used as the reconstructed side information of the residual feature map. The side information can be obtained by further performing feature extraction on the residual feature map of the enhancement layer of the target region and can be regarded as a feature map having the same dimensions as the aforementioned residual feature map. Therefore, the side information extraction network is configured to further perform feature extraction on the residual feature map of the enhancement layer of the target region and obtain a feature map having the same dimensions as the aforementioned residual feature map.

[0230] The side information processing network can perform feature extraction on the side information and output a first feature map having the same resolution as the residual feature map. For example, in the side information processing network, a neural network formed by interpolating and cascading three deconvolution layers and two activation layers is used to implement the above function.

[0231] See the above description of the probability estimation network, and the details will not be explained again here.

[0232] b. If the correction information is multiple pixel values, input (estimate) the multiple pixel values ​​into a feature estimation network to obtain a second feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and then input the first feature map and the second feature map into a probability estimation network to obtain multiple probability distributions.

[0233] The feature estimation network can convert pixel values ​​(multiple pixel values) represented by pixel regions into feature values ​​(second feature map) represented by feature regions. The feature estimation network can use the structure of the probability estimation network shown in FIG. 11b, and is different from the probability estimation network in the input process, output process, and training process. Due to the differences in the input process, output process, and training process, even if the network structure is the same (i.e., the included layer structure is the same), the feature estimation network can still be considered as a different network and can complete different functions.

[0234] For the side information processing network and the probability estimation network, please refer to the above description, and the details will not be described again here.

[0235] 2. Obtain multiple probability distributions based on the reconstructed side information of the residual feature maps

[0236] 2.1. Input the reconstructed side information into a probability estimation network to obtain multiple probability distributions.

[0237] See the above description of the probability estimation network, and the details will not be explained again here.

[0238] 2.2. Obtain multiple probability distributions based on the reconstructed side information and the reconstructed pixels

[0239] In this embodiment of the present invention, the reconstructed pixels of the residual feature map may be input into a feature estimation network to obtain a third feature map, the reconstructed side information may be input into a side information processing network to obtain a first feature map, and the first feature map and the third feature map may be input into a probability estimation network to obtain a plurality of probability distributions.

[0240] For the feature estimation network, the side information processing network, and the probability estimation network, please refer to the above descriptions, and the details will not be described again here.

[0241] In this embodiment of the present application, AI encoding is applied to the enhancement layer of a selected target region. Based on this, the correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then encodes the residual feature map of the enhancement layer of the target region based on the correction information. The encoding of the enhancement layer is performed only for the necessary region (target region), which can reduce the encoding complexity of the enhancement layer and improve the encoding efficiency of the enhancement layer. In addition, the encoding is performed based on the correction information, which can improve the encoding accuracy.

[0242] 12 is a flowchart of a process 1200 of a method for decoding a region enhancement layer according to one embodiment of the present application. The process 1200 may be performed by the video decoder 30. The process 1200 is described as a series of steps or operations. It should be understood that the steps or operations of the process 1200 may be performed in various orders and / or simultaneously and are not limited to the order of execution shown in FIG. 12. Assuming a bitstream is used by the video decoder 30, the process 1200, which includes the following steps, is performed to decode the bitstream and obtain reconstructed pixels of the region enhancement layer. The process 1200 may include the following steps:

[0243] Step 1201: Obtain the reconstructed pixels of the base layer of the target region.

[0244] Corresponding to the encoder side, the decoder side may receive a bitstream from the encoder side, which includes a base layer bitstream obtained after the encoder side encodes the original pixels of the image, and then the decoder side decodes the base layer bitstream to obtain a reconstructed image of the base layer of the image.

[0245] In this embodiment of the present application, the decoder side can obtain the reconstructed pixels of the base layer of the target region in the same manner as that of the encoder side. For details, see the description of step 801. The details will not be described again in this specification.

[0246] Step 1202: Input the reconstructed pixels into a correction network to obtain correction information for the target region.

[0247] For step 1202, please refer to the description of step 802. The details will not be described again here.

[0248] Step 1203: Obtain the enhancement layer bitstream of the target region.

[0249] In step 804, the encoder side encodes the residual feature map to obtain the enhancement layer bitstream of the target region, and accordingly, the bitstream received by the decoder side further includes the enhancement layer bitstream of the target region.

[0250] Step 1204: Decode the enhancement layer bitstream to obtain the enhancement layer residual feature map of the target region.

[0251] In this embodiment of the present application, multiple probability distributions may be obtained, where the multiple probability distributions correspond to multiple feature value bitstreams included in the enhancement layer bitstream, and then entropy decoding is performed on the corresponding feature value bitstreams in the enhancement layer bitstream based on the multiple probability distributions to obtain a residual feature map of the target region.

[0252] The enhancement layer bitstream includes multiple feature value bitstreams. To perform entropy decoding on the multiple feature value bitstreams, probability distributions for the multiple feature value bitstreams need to be obtained. For some methods for obtaining the multiple probability distributions, see the description of step 804. The details will not be described again in this specification.

[0253] Step 1205: Input the residual feature map and correction information into a decoding network to obtain the reconstructed pixels of the enhancement layer of the target region.

[0254] Corresponding to step 803, the encoder side obtains the residual feature map of the enhancement layer of the target region through an encoding network based on the input correction information and the original pixels of the target region, and the decoder side obtains the reconstructed pixels of the enhancement layer of the target region through a decoding network based on the input residual feature map and correction information.

[0255] The correction information can have two cases: one is multiple pixel values, and the other is multiple feature values. Correspondingly, the decoding network can also use two structures. In this embodiment of the present application, the decoding network (Decoder) on the decoder side inputs the correction information and residual feature map of the enhancement layer of the target region, and outputs the reconstructed pixels of the enhancement layer of the target region.

[0256] 13a is an exemplary diagram of a decoding network. As shown in FIG. 13a, the decoding network may be a first decoding network including four convolutional layers and three GDNs. The residual feature map is first input to the first decoding network to obtain reconstructed residual pixels of the enhancement layer of the target region, and then the reconstructed residual pixels and corresponding pixel values ​​in the correction information are summed to obtain reconstructed pixels, where the correction information is a plurality of pixel values. In this embodiment, since the correction information (a plurality of pixel values) corresponds to a pixel region, the correction information and the reconstructed residual pixels of the enhancement layer of the target region may be directly added.

[0257] 13b is an exemplary diagram of a decoding network. As shown in FIG. 13b, the decoding network may be a second decoding network including four convolutional layers and three GDNs. The residual feature map is first input to the second decoding network, and the output of any convolutional layer (e.g., the second convolutional layer) in the second decoding network is summed with the corresponding feature value (multiple feature values) in the correction information. The summation result is then input to a network layer after any convolutional layer (e.g., the second convolutional layer) in the second decoding network to obtain a reconstructed pixel. In this embodiment, since the correction information (multiple feature values) corresponds to a feature region, the residual feature map needs to be first input to the second decoding network to be converted into the feature region, and then the feature region and the multiple feature values ​​are summed.

[0258] It should be noted that in addition to the above two examples of the decoding network, a decoding network of another structure may alternatively be used in the embodiments of the present application, which is not particularly limited herein.

[0259] In this embodiment of the present application, AI decoding is applied to the enhancement layer bitstream. Based on this, a correction network removes noise signals that are not beneficial to the encoding of the AI ​​enhancement layer from the reconstructed pixels of the base layer to obtain correction information, and then decodes the enhancement layer bitstream based on the correction information. The enhancement layer decoding is performed only for the necessary region (target region). This can reduce the decoding complexity of the enhancement layer and improve the decoding efficiency of the enhancement layer. In addition, decoding is performed based on the correction information. This can improve the decoding accuracy.

[0260] Below, some specific embodiments are provided to illustrate the solutions of the embodiments of the present application.

[0261] Example 1 An exemplary diagram of the encoding and decoding process is shown in Figure 14. As shown in Figure 14, the process in this embodiment is described below.

[0262] Encoder side:

[0263] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0264] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, such as the H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0265] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0266] The determining method may include selecting a region with complex texture, selecting a region that is interesting to the human eye, or randomly selecting one or more regions. Related techniques may be applied in this step, which is not particularly limited in this specification.

[0267] For example, if the variance of a region is greater than a threshold t1, the region is determined to have a complex texture and can be used as a target region. Alternatively, if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region is determined to have a complex texture and can be used as a target region. t1 and a are numbers greater than 0, and t2 is a number between 0 and 1.

[0268] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0269] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information includes the above-mentioned multiple pixel domains or the above-mentioned multiple feature domains. For example, the following three methods may be used:

[0270] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in FIG. 9a, and the pixel values ​​are output as p.

[0271] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9b, and multiple feature values ​​are output as p.

[0272] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0273] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0274] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. When the correction information is pixel values, the encoder 2 uses the structure shown in Figure 10a. When the correction information is feature values, the encoder 2 uses the structure shown in Figure 10b.

[0275] It should be noted that the specific quantization method is not limited in the embodiments of the present application. Alternatively, the floating-point number in y may be truncated to obtain an integer, or the quantized feature value may be obtained by performing quantization based on a preset quantization step.

[0276] 5. p is input to the probability estimation network, and y ^ The probability distribution of all feature values ​​of y is output, and then, based on the probability distribution, ^Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0277] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be input to the probability estimation network, but the structure of the probability estimation network is diverse. When the correction information is pixel values, the probability estimation network uses the structure shown in FIG. 11a. When the correction information is feature values, the probability estimation network uses the structure shown in FIG. 11b.

[0278] Decoder side:

[0279] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain the reconstructed image xc.

[0280] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0281] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0282] 4. p is input to the probability estimation network, and y ^ Then, based on the probability distribution, entropy decoding (AD) is performed on the enhancement layer bitstream (Bitstream2) to obtain the enhancement layer residual feature map y ^ Then, get y ^ Inverse quantization (IQ) is performed on the ~ ) to get the

[0283] 5. p and y ~is input to the enhancement layer decoder (Decoder2), and the reconstructed pixels (xd) of the enhancement layer of xc1 are output.

[0284] 6. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0285] Example 2 15 is an exemplary diagram of the encoding and decoding process. As shown in FIG. 15, the difference between this embodiment and embodiment 1 is in how to obtain the probability distribution.

[0286] Encoder side:

[0287] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0288] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, such as the H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0289] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0290] The determining method may include selecting a region with complex texture, selecting a region that is interesting to the human eye, or randomly selecting one or more regions. Related techniques may be applied in this step, which is not particularly limited in this specification.

[0291] For example, if the variance of a region is greater than a threshold t1, the region is a region with complex texture and may be used as a target region, or if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region may be determined as a region with complex texture and may be used as a target region, where t1 and a are numbers greater than 0 and t2 is a number between 0 and 1.

[0292] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0293] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information is multiple pixel domains or multiple feature domains. For example, the following three methods may be used:

[0294] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9a, and multiple pixel values ​​are output as p.

[0295] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9b, and multiple feature values ​​are output as p.

[0296] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0297] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0298] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. When the correction information is pixel values, the encoder 2 uses the structure shown in Figure 10a. When the correction information is feature values, the encoder 2 uses the structure shown in Figure 10b.

[0299] It should be noted that the specific quantization method is not limited in the embodiments of the present application. Alternatively, the floating-point number in y may be truncated to obtain an integer, or the quantized feature value may be obtained by performing quantization based on a preset quantization step.

[0300] 5. y is input to the side information extraction network to obtain the side information z of y, and quantization (Q) is performed on z to obtain the quantized side information (z ^ ) and z ^ Entropy coding (AE) is performed on Bitstream 3 to obtain the bitstream (Bitstream 3), and then entropy decoding (AD) is performed on Bitstream 3 to obtain the reconstructed side information (z ^ ) In addition, the encoder side can send bitstream 3 to the decoder side.

[0301] 6.z ^ is input to the probability estimation network, and y ^ The probability distribution of all feature values ​​of y is output, and then, based on the probability distribution, ^Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0302] Decoder side:

[0303] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain the reconstructed image xc.

[0304] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0305] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0306] 4. The bitstream (Bitstream3) is parsed and the reconstructed side information (z ^ ) to get the

[0307] 5.z ^ is input to the probability estimation network, which outputs the probability distribution of all feature bitstreams in the enhancement layer bitstream (Bitstream2), and then performs entropy coding (AE) on Bitstream2 based on the probability distribution to obtain the enhancement layer residual feature map (y ^ ) to get the

[0308] 6. p and y ~ is input to the enhancement layer decoder (Decoder2), and the reconstructed pixels (xd) of the enhancement layer of xc1 are output.

[0309] 7. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0310] Example 3 16 is an exemplary diagram of the encoding and decoding process. As shown in FIG. 16, the difference between this embodiment and embodiment 2 is in how to obtain the probability distribution.

[0311] Encoder side:

[0312] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0313] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, such as the H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0314] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0315] The determining method may include selecting a region with complex texture, selecting a region that is interesting to the human eye, or randomly selecting one or more regions. Related techniques may be applied in this step, which is not particularly limited in this specification.

[0316] For example, if the variance of a region is greater than a threshold t1, the region is a region with complex texture and may be used as a target region, or if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region may be determined as a region with complex texture and may be used as a target region, where t1 and a are numbers greater than 0 and t2 is a number between 0 and 1.

[0317] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0318] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information is multiple pixel domains or multiple feature domains. For example, the following three methods may be used:

[0319] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9a, and multiple pixel values ​​are output as p.

[0320] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9b, and multiple feature values ​​are output as p.

[0321] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0322] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0323] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. When the correction information is pixel values, the encoder 2 uses the structure shown in Figure 10a. When the correction information is feature values, the encoder 2 uses the structure shown in Figure 10b.

[0324] It should be noted that the specific quantization method is not limited in the embodiments of the present application. Alternatively, the floating-point number in y may be truncated to obtain an integer, or the quantized feature value may be obtained by performing quantization based on a preset quantization step.

[0325] 5. y is input to the side information extraction network to obtain the side information z of y, and quantization (Q) is performed on z to obtain the quantized side information (z ^ ) and z ^ Entropy coding (AE) is performed on Bitstream 3 to obtain the bitstream (Bitstream 3), and then entropy decoding (AD) is performed on Bitstream 3 to obtain the reconstructed side information (z ^ ) In addition, the encoder side can send bitstream 3 to the decoder side.

[0326] 6.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0327] 7. If p is multiple pixel values, then multiple pixel values ​​need to be input to the feature estimation network for feature extraction. If p is multiple feature values, then multiple feature values ​​do not need to be input to the feature estimation network.

[0328] 8. The reconstructed side information of the feature domain and p of the feature domain are input to the probability estimation network, and y^ The probability distribution of all feature values ​​of y is output, and then, based on the probability distribution, ^ Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0329] Decoder side:

[0330] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain the reconstructed image xc.

[0331] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0332] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0333] 4. The bitstream (Bitstream3) is parsed and the reconstructed side information (z ^ ) to get the

[0334] 5.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0335] 6. If p is multiple pixel values, then multiple pixel values ​​need to be input to the feature estimation network for feature extraction. If p is multiple feature values, then multiple feature values ​​do not need to be input to the feature estimation network.

[0336] 7. The reconstructed side information of the feature domain and p of the feature domain are input to the probability estimation network, which outputs the probability distribution of all feature bitstreams in the enhancement layer bitstream (Bitstream2). Then, based on the probability distribution, entropy coding (AE) is performed on Bitstream2 to obtain the enhancement layer residual feature map (y ^ ) to get the

[0337] 8. p and y ~ is input to the enhancement layer decoder (Decoder2), and the reconstructed pixels (xd) of the enhancement layer of xc1 are output.

[0338] 9. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0339] Example 4 17 is an exemplary diagram of the encoding and decoding process. As shown in FIG. 17, the difference between this embodiment and embodiment 3 is in how to obtain the probability distribution.

[0340] Encoder side:

[0341] 1. A base layer encoder (Encoder1) encodes an original image x to obtain a base layer bitstream (Bitstream1), and a base layer decoder (Decoder1) decodes Bitstream1 to obtain a reconstructed image xc.

[0342] The encoder 1 and the decoder 1 may perform base layer encoding and decoding by using a conventional video coding standard, such as the H.264 / AVC, H.265 / HEVC, or H.266 / VVC standard, or may perform base layer encoding and decoding by using an existing JPEG image encoding standard, which is not particularly limited in this specification.

[0343] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0344] The determining method may include selecting a region with complex texture, selecting a region that is interesting to the human eye, or randomly selecting one or more regions. Related techniques may be applied in this step, which is not particularly limited in this specification.

[0345] For example, if the variance of a region is greater than a threshold t1, the region is a region with complex texture and may be used as a target region, or if the gradient is greater than a threshold a and the variance in the region is greater than t2, the region may be determined as a region with complex texture and may be used as a target region, where t1 and a are numbers greater than 0 and t2 is a number between 0 and 1.

[0346] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0347] The correction network functions to remove noise signals that are not beneficial to the encoding of the AI ​​enhancement layer. The correction network may output multiple pixel domains and / or multiple feature domains. The correction information is multiple pixel domains or multiple feature domains. For example, the following three methods may be used:

[0348] Method 1: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9a, and multiple pixel values ​​are output as p.

[0349] Method 2: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9b, and multiple feature values ​​are output as p.

[0350] Method 3: The reconstructed pixels of the base layer of xc1 are input to the correction network shown in Figure 9c, multiple pixel values ​​and multiple feature values ​​are output, and one of the pixel values ​​and feature values ​​is used as p.

[0351] 4. x and p are input to the enhancement layer encoder (Encoder2), which outputs the enhancement layer residual feature map (y) of the target region, and then quantization (Q) is performed on y to obtain the quantized residual feature map (y ^ ) is obtained.

[0352] There are two kinds of correction information: pixel values ​​and feature values. Both pixel values ​​and feature values ​​can be used as input to the encoder 2, but the structure of the encoder 2 changes. When the correction information is pixel values, the encoder 2 uses the structure shown in Figure 10a. When the correction information is feature values, the encoder 2 uses the structure shown in Figure 10b.

[0353] It should be noted that the specific quantization method is not limited in the embodiments of the present application. Alternatively, the floating-point number in y may be truncated to obtain an integer, or the quantized feature value may be obtained by performing quantization based on a preset quantization step.

[0354] 5. y is input to the side information extraction network to obtain the side information z of y, and quantization (Q) is performed on z to obtain the quantized side information (z ^ ) and z ^Entropy coding (AE) is performed on Bitstream 3 to obtain the bitstream (Bitstream 3), and then entropy decoding (AD) is performed on Bitstream 3 to obtain the reconstructed side information (z ^ ) In addition, the encoder side can send bitstream 3 to the decoder side.

[0355] 6.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0356] 7.xc1 is input to the feature estimation network for feature extraction.

[0357] 8. The reconstructed side information and xc1, both of which belong to the feature domain, are input to the probability estimation network, and y ^ The probability distribution of all feature values ​​of y is output, and then, based on the probability distribution, ^ Entropy coding (AE) is performed on it to obtain the enhancement layer bitstream (Bitstream2).

[0358] Decoder side:

[0359] 1. The base layer decoder (Decoder1) parses the base layer bitstream (Bitstream1) to obtain the reconstructed image xc.

[0360] 2. At least one region to be strengthened is determined in xc, where one of the regions is a target region xc1.

[0361] 3. The reconstructed pixels of the base layer of xc1 are input to the correction network, which outputs the correction information (p).

[0362] 4. The bitstream (Bitstream3) is parsed and the reconstructed side information (z^ ) to get the

[0363] 5.z ^ is input into the side information processing network to obtain the reconstructed side information of the feature region.

[0364] 6.xc1 is input to the feature estimation network for feature extraction.

[0365] 7. The reconstructed side information and xc1, both of which belong to the feature domain, are input to a probability estimation network, which outputs the probability distribution of all feature bitstreams in the enhancement layer bitstream (Bitstream2). Then, based on the probability distribution, entropy coding (AE) is performed on Bitstream2 to obtain the enhancement layer residual feature map (y ^ ) to get the

[0366] 8. p and y ~ is input to the enhancement layer decoder (Decoder2), and the reconstructed pixels (xd) of the enhancement layer of xc1 are output.

[0367] 9. The pixel corresponding to the position of xc1 in xc is replaced by pixel xd to obtain an enhanced reconstructed image.

[0368] 18 is an exemplary diagram of the structure of an encoding device 1800 according to an embodiment of the present application. As shown in FIG. 18, the encoding device 1800 of this embodiment may be used on the encoder 20 side. The encoding device 1800 may include an acquiring module 1801, a processing module 1802, and an encoding module 1803.

[0369] The acquisition module 1801 is configured to acquire reconstructed pixels of a base layer for a target region. The processing module 1802 is configured to input the reconstructed pixels to a correction network to obtain correction information for the target region, and to input the correction information and original pixels of the target region to an encoding network to obtain an enhancement layer residual feature map for the target region. The encoding module 1803 is configured to encode the residual feature map to obtain an enhancement layer bitstream for the target region.

[0370] In a possible implementation, the processing module 1802 is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area, where the correction information is a plurality of pixel values ​​or a plurality of feature values.

[0371] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0372] In a possible implementation, the encoding module 1803 is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0373] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map.

[0374] In a possible implementation, when the correction information is a plurality of feature values, the encoding module 1803 is specifically configured to input the reconstructed side information into a side information processing network to obtain a first feature map, and to input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0375] In a possible implementation, the encoding module 1803 is specifically configured, when the correction information is a plurality of pixel values, to input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, to input the reconstructed side information into a side information processing network to obtain a first feature map, and to input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0376] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature values ​​included in the residual feature map, and to perform entropy coding on the corresponding feature values ​​in the residual feature map based on the plurality of probability distributions, respectively, to obtain an enhancement layer bitstream.

[0377] In a possible implementation, the encoding module 1803 is configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0378] In a possible implementation, the encoding module 1803 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0379] In a possible implementation, the encoding module 1803 is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0380] In a possible implementation, the encoding module 1803 is further configured to input the residual feature map to a side information extraction network to obtain side information of the residual feature map, and then perform entropy coding on the side information or the quantized side information, and write the encoded side information into a bitstream.

[0381] In a possible implementation, the encoding network includes a first encoding network, and the encoding module 1803 is configured to, when the correction information is a plurality of pixel values, perform subtraction on corresponding pixel values ​​in the original pixel and the correction information, and input the subtraction result into the first encoding network to obtain a residual feature map.

[0382] In a possible implementation, the encoding network includes a second encoding network, and the encoding module 1803 is configured to input original pixels to the second encoding network, and if the correction information is a plurality of feature values, perform subtraction on the output of any convolutional layer in the second encoding network and the corresponding feature values ​​in the correction information, and input the subtraction result to a network layer after the any convolutional layer in the second encoding network to obtain a residual feature map.

[0383] In a possible implementation, the encoding module 1803 is further configured to encode the image to which the target region belongs to obtain a base layer bitstream of the image, decode the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0384] In a possible implementation, the encoding module 1803 is specifically configured to divide the reconstructed image to obtain a plurality of regions, and determine a region in the plurality of regions whose variance is greater than a first threshold as at least one region, or determine a proportion of pixels in each of the plurality of regions whose gradient is greater than a second threshold and is present in the total pixels, and determine a region in which the proportion is greater than a third threshold as at least one region.

[0385] In a possible implementation, the encoding module 1803 is further configured to use side information of the residual feature map as reconstructed side information of the residual feature map.

[0386] The apparatus in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 8, and the implementation principle and technical effect of the apparatus are similar and will not be described again in this specification.

[0387] 19 is an exemplary diagram of the structure of a decoding device 1900 according to an embodiment of the present application. As shown in FIG. 19, the decoding device 1900 in this embodiment may be used in the decoder side 30. The decoding device 1900 may include an obtaining module 1901, a processing module 1902, and a decoding module 1903.

[0388] The acquisition module 1901 is configured to acquire reconstructed pixels of a base layer for a target region. The processing module 1902 is configured to input the reconstructed pixels into a correction network to obtain correction information for the target region. The acquisition module 1901 is further configured to acquire an enhancement layer bitstream for the target region. The decoding module 1903 is configured to decode the enhancement layer bitstream to obtain an enhancement layer residual feature map for the target region. The processing module 1902 is further configured to input the residual feature map and the correction information into the decoding network to obtain reconstructed pixels of the enhancement layer for the target region.

[0389] In a possible implementation, the processing module 1902 is specifically configured to input the reconstructed pixels into a correction network to obtain at least one of a plurality of pixel values ​​and a plurality of feature values ​​of the target area, where the correction information is a plurality of pixel values ​​or a plurality of feature values.

[0390] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the correction information, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and to perform entropy decoding on the corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain a residual feature map.

[0391] In a possible implementation, the decoding module 1903 is specifically configured to input the correction information into a probability estimation network to obtain a plurality of probability distributions.

[0392] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the correction information and the reconstructed side information of the residual feature map.

[0393] In a possible implementation, the decoding module 1903 is specifically configured to, when the correction information is a plurality of feature values, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the plurality of feature values ​​and the first feature map into a probability estimation network to obtain a plurality of probability distributions.

[0394] In a possible implementation, the decoding module 1903 is specifically configured, when the correction information is a plurality of pixel values, to input the plurality of pixel values ​​into a feature estimation network to obtain a second feature map, to input the reconstructed side information into a side information processing network to obtain a first feature map, and to input the first feature map and the second feature map into a probability estimation network to obtain a plurality of probability distributions.

[0395] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information of the residual feature map, where the plurality of probability distributions correspond to a plurality of feature value bitstreams included in the enhancement layer bitstream, and to perform entropy decoding on the corresponding feature value bitstreams in the enhancement layer bitstream based on the plurality of probability distributions, respectively, to obtain the residual feature map.

[0396] In a possible implementation, the decoding module 1903 is specifically configured to input the reconstructed side information into a probability estimation network to obtain a plurality of probability distributions.

[0397] In a possible implementation, the decoding module 1903 is specifically configured to obtain a plurality of probability distributions based on the reconstructed side information and the reconstructed pixels.

[0398] In a possible implementation, the decoding module 1903 is specifically configured to input the reconstructed pixels into a feature estimation network to obtain a third feature map, input the reconstructed side information into a side information processing network to obtain a first feature map, and input the first feature map and the third feature map into a probability estimation network to obtain a plurality of probability distributions.

[0399] In a possible implementation, the decoding module 1903 is further configured to input the residual feature map into a side information extraction network to obtain side information of the residual feature map, and use the side information as reconstructed side information of the residual feature map.

[0400] In a possible implementation, the decoding module 1903 is further configured to obtain a side information bitstream of the target region and to parse the side information bitstream to obtain the reconstructed side information.

[0401] In a possible implementation, the decoding network includes a first decoding network, and the decoding module 1903 is specifically configured to input the residual feature map into the first decoding network to obtain a reconstructed residual pixel of the enhancement layer of the target region, and, when the correction information is a plurality of pixel values, sum the reconstructed residual pixel and the corresponding pixel value in the correction information to obtain a reconstructed pixel.

[0402] In a possible implementation, the decoding network includes a second decoding network, and the decoding module 1903 is specifically configured to input the residual feature map into the second decoding network, and if the correction information is a plurality of feature values, sum the output of any convolutional layer in the second decoding network with the corresponding feature value in the correction information, and input the summation result into a network layer after the any convolutional layer in the second decoding network to obtain a reconstructed pixel.

[0403] In a possible implementation, the decoding module 1903 is further configured to obtain a base layer bitstream of the image to which the target region belongs, parse the base layer bitstream to obtain a reconstructed image of the base layer of the image, and determine at least one region to be enhanced based on the reconstructed image, where the target region is one of the at least one region.

[0404] In a possible implementation, the decoding module 1903 is specifically configured to divide the reconstructed image to obtain a plurality of regions, and determine a region in the plurality of regions whose variance is greater than a first threshold as at least one region, or determine a proportion of pixels in each of the plurality of regions whose gradient is greater than a second threshold and is present in the total pixels, and determine a region in which the proportion is greater than a third threshold as at least one region.

[0405] The apparatus in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 12, and the implementation principle and technical effect of the apparatus are similar and will not be described again in this specification.

[0406] In the implementation process, the steps in the above-described method embodiments can be implemented by using hardware integrated logic circuits in a processor or by using instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly presented as being performed and completed by a hardware encoding processor, or can be performed and completed by a combination of hardware modules and software modules in the encoding processor. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned method in combination with the processor's hardware.

[0407] The memory in the above embodiments may be volatile or nonvolatile memory, or may include both volatile and nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM) and used as an external cache. By way of example and not limitation, many forms of RAM may be used, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchlink dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus random access memory (direct rambus RAM, DR RAM). It should be noted that memory in the systems and methods described herein includes, but is not limited to, these and any other suitable types of memory.

[0408] Those skilled in the art can recognize that, in combination with the examples described in the embodiments disclosed herein, the units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but this implementation should not be considered to go beyond the scope of this application.

[0409] Those skilled in the art will clearly understand that for convenience of description, the detailed operation processes of the above-mentioned systems, devices and units may be referred to the corresponding processes in the above-mentioned method embodiments, and the details will not be described again in this specification.

[0410] In some embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the described device embodiments are merely examples. For example, the division into units is merely a logical division of function, and actual implementation may involve other divisions. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. In addition, the shown or discussed mutual couplings, direct couplings, or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0411] The units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, located in one location, or distributed over multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0412] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, and each of the units may exist physically alone, or two or more units may be integrated into one unit.

[0413] When a function is implemented in the form of a software functional unit and sold or used as an independent product, the function may be stored in a computer-readable storage medium. Based on this understanding, essentially, the technical solution in the present application, or a portion of the technical solution, or a portion of the technical solution, may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for instructing a computer device (such as a personal computer, a server, a network device, etc.) to execute all or part of the steps of the method in the embodiments of the present application. The above-mentioned storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0414] The above description is merely a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any variations or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application shall fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. It is an encoding method, This is the step of encoding the original image. Steps to obtain the first bitstream, The step is to determine the target region based on the reconstructed image obtained by decoding the first bitstream, The aforementioned target region is the region to be strengthened, step, This step involves inputting the reconstructed pixels obtained from the target region in the reconstructed image into the first neural network. A step of obtaining correction information for the target region, The step involves inputting the correction information and original pixels obtained from the target region in the original image into a second neural network. The steps include obtaining a residual feature map of the target region, This is a step of encoding the residual feature map, The step of obtaining the second bitstream, Methods that include...

2. The correction information includes at least one of a plurality of pixel values ​​and a plurality of feature values ​​in the target region. The method according to claim 1.

3. The step involves encoding the residual feature map and obtaining a second bitstream. Steps include obtaining multiple probability distributions based on the correction information, where the multiple probability distributions correspond to the multiple feature values ​​included in the residual feature map, The steps include: performing entropy coding on the corresponding feature values ​​in the residual feature map based on the respective multiple probability distributions, and obtaining the second bitstream; The method according to claim 2, including the method described in claim 2.

4. The step of obtaining multiple probability distributions based on the correction information is: The step involves inputting the correction information into the third neural network, and obtaining the multiple probability distributions. The method according to claim 3, including the method described in claim 3.

5. The step of obtaining multiple probability distributions based on the correction information is: A step of obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map, The method according to claim 3, including the method described in claim 3.

6. The step of obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map is as follows: The step of inputting the reconstructed side information into the fourth neural network when the correction information is the plurality of feature values, and obtaining the first feature map, The step involves inputting the aforementioned multiple feature values ​​and the first feature map into a third neural network, and obtaining the aforementioned multiple probability distributions. The method according to claim 5, including the method described in claim 5.

7. The step of obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map is as follows: If the correction information is the plurality of pixel values, the step is to input the plurality of pixel values ​​to the fifth neural network and obtain the second feature map. The steps include inputting the reconstructed side information into the fourth neural network, obtaining the first feature map, and The steps include inputting the first feature map and the second feature map into a third neural network, and obtaining the plurality of probability distributions, The method according to claim 5, including the method described in claim 5.

8. The step involves encoding the residual feature map and obtaining a second bitstream. The step of obtaining multiple probability distributions based on the reconfigured side information of the residual feature map, wherein the multiple probability distributions correspond to the multiple feature values ​​included in the residual feature map. The steps include: performing entropy coding on the corresponding feature values ​​in the residual feature map based on the respective multiple probability distributions, and obtaining the second bitstream; The method according to claim 2, including the method described in claim 2.

9. A decoding method, The steps include obtaining the first bitstream of the original image, The step of determining a target region based on a first reconstructed image obtained by decoding the first bitstream, The aforementioned target region is the region to be strengthened, step, The first step involves inputting a first set of reconstructed pixels obtained from the target region in the first reconstructed image into a first neural network, and obtaining correction information for the target region. The steps include obtaining the second bitstream, The step of decoding the second bitstream, and obtaining the residual feature map of the target region, The steps include inputting the residual feature map and the correction information into a second neural network, and obtaining a second set of reconstructed pixels in the target region. The steps include replacing the first pixel in the first set of reconstructed pixels with the second pixel in the second set of reconstructed pixels, obtaining a second reconstructed image, wherein the first and second pixels have corresponding positions, and Methods that include...

10. The correction information includes at least one of a plurality of pixel values ​​and a plurality of feature values ​​in the target region. The method according to claim 9, including the method described in claim 9.

11. The step of decoding the second bitstream, wherein the step of obtaining the residual feature map of the target region is, The step of obtaining multiple probability distributions based on the correction information, wherein the multiple probability distributions correspond to multiple feature value bitstreams included in the second bitstream, The steps include: performing entropy decoding on the corresponding feature value bitstream in the second bitstream based on the plurality of probability distributions, and obtaining the residual feature map; The method according to claim 9, including the method described in claim 9.

12. The step of obtaining multiple probability distributions based on the correction information is as follows: The step involves inputting the correction information into the third neural network, and obtaining the multiple probability distributions. The method according to claim 11, including the method described in claim 11.

13. The step of obtaining multiple probability distributions based on the correction information is as follows: A step of obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map, The method according to claim 11, including the method described in claim 11.

14. The step of obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map is as follows: If the correction information consists of multiple feature values, the step is to input the reconstructed side information into the fourth neural network, and to obtain the first feature map. The step involves inputting the aforementioned multiple feature values ​​and the first feature map into a third neural network, and obtaining the aforementioned multiple probability distributions. The method according to claim 13, including the method described in claim 13.

15. The step of obtaining the plurality of probability distributions based on the correction information and reconstructed side information of the residual feature map is as follows: If the correction information consists of multiple pixel values, the step is to input the multiple pixel values ​​into the fifth neural network and obtain a second feature map. The steps include inputting the reconstructed side information into the fourth neural network, obtaining the first feature map, and The steps include inputting the first feature map and the second feature map into a third neural network, and obtaining the plurality of probability distributions, The method according to claim 13, including the method described in claim 13.

16. The step involves decoding the second bitstream and obtaining a residual feature map of the target region. The step of obtaining multiple probability distributions based on the reconfigured side information of the residual feature map, wherein the multiple probability distributions correspond to multiple feature value bitstreams included in the second bitstream. The steps include: performing entropy decoding on the corresponding feature value bitstream in the second bitstream based on the plurality of probability distributions, and obtaining the residual feature map; The method according to claim 9, including the method described in claim 9.

17. It is an encoder, A processing circuit for carrying out the method according to any one of claims 1 to 8, or, One or more processors, Includes memory configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors become capable of carrying out the method according to any one of claims 1 to 8. Encoder.

18. It is a decoder, Includes a processing circuit for carrying out the method described in any one of claims 9 to 16, or, One or more processors, Includes memory configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors become capable of carrying out the method described in any one of claims 9 to 16. decoder.

19. A computer-readable storage medium on which computer programs are stored, When the computer program is executed on the computer, the computer becomes capable of performing the method according to any one of claims 1 to 16. Computer-readable storage medium.

20. A computer program that includes instructions, When the instruction is executed on the computer or processor, the computer or processor becomes capable of carrying out the method according to any one of claims 1 to 16. Computer program.

21. A computer-readable storage medium comprising a first bitstream and a second bitstream, The first bitstream and the second bitstream are obtained according to the method described in any one of claims 1 to 8. Computer-readable storage medium.

Images