Coding method, coder and storage medium

Abstract

Embodiments of the present application provide a coding method, a codec and a storage medium. The decoding method comprises: parsing a code stream, determining a first reconstructed image; determining a first residual image corresponding to the first reconstructed image according to a neural network; adjusting residual values in the first residual image to determine a second residual image; and determining a second reconstructed image according to the first reconstructed image and the second residual image. Embodiments of the present application do not directly use residual information determined based on a neural network to recover a reconstructed image, but use the residual information after residual adjustment to recover the reconstructed image, which helps to correct the output error of the neural network, thereby improving the coding performance.

Description

Coding and decoding method, codec and storage medium Technical Field

[0001] The present application relates to the technical field of video coding and decoding, and in particular to a coding and decoding method, a codec, and a storage medium. Background Art

[0002] With the development of deep learning technology, neural network-based encoding and decoding tools have been introduced into video encoding and decoding frameworks, such as neural network-based super resolution (NNSR) and neural network-based post filter (NNPF). Most of these neural networks are based on residual network structures, enabling them to predict residual information in images through training. However, the neural network training process and the actual encoding and testing process are not completely consistent. For example, the training data for the neural network and the actual test data are often different. This inconsistency can lead to errors in the neural network's output information, thereby reducing encoding and decoding performance.

[0003] Summary of the Invention

[0004] The embodiments of the present application provide a coding and decoding method, a codec, and a storage medium. The following introduces various aspects involved in the present application.

[0005] In a first aspect, a decoding method is provided, which is applied to a decoder, including: parsing a code stream to determine a first reconstructed image; determining a first residual image corresponding to the first reconstructed image based on a neural network; adjusting the residual value in the first residual image to determine a second residual image; and determining a second reconstructed image based on the first reconstructed image and the second residual image.

[0006] In a second aspect, a coding method is provided, which is applied to an encoder, including: determining a first reconstructed image; determining a first residual image corresponding to the first reconstructed image based on a neural network; adjusting the residual value in the first residual image to determine a second residual image; and determining a second reconstructed image based on the first reconstructed image and the second residual image.

[0007] According to a third aspect, a decoder is provided, comprising: a first determination unit configured to parse a code stream and determine a first reconstructed image; a second determination unit configured to determine a first residual image corresponding to the first reconstructed image based on a neural network; a third determination unit configured to adjust the residual value in the first residual image and determine a second residual image; and a fourth determination unit configured to determine a second reconstructed image based on the first reconstructed image and the second residual image.

[0008] In a fourth aspect, a decoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method described in the first aspect when running the computer program.

[0009] In a fifth aspect, an encoder is provided, comprising: a first determination unit configured to determine a first reconstructed image; a second determination unit configured to determine a first residual image corresponding to the first reconstructed image based on a neural network; a third determination unit configured to adjust the residual value in the first residual image to determine a second residual image; and a fourth determination unit configured to determine a second reconstructed image based on the first reconstructed image and the second residual image.

[0010] In a sixth aspect, an encoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method described in the second aspect when running the computer program.

[0011] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the method as described in the first aspect or the second aspect is implemented.

[0012] In an eighth aspect, a non-volatile computer-readable storage medium for storing a bit stream is provided, wherein the bit stream is generated by an encoding method using an encoder, or the bit stream is decoded by a decoding method using a decoder, wherein the decoding method is the method described in the first aspect, and the encoding method is the method described in the second aspect.

[0013] In a ninth aspect, a computer program product is provided, comprising a computer program, which implements the method described in the first aspect or the second aspect when executed.

[0014] The embodiment of the present application does not directly use the residual information determined based on the neural network (i.e., the first residual image mentioned above) to restore and reconstruct the image, but performs residual adjustment on the residual information before using it to restore and reconstruct the image. This helps to correct the output error of the neural network, thereby improving the encoding and decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a structural diagram illustrating an example of a video encoder to which an embodiment of the present application may be applied.

[0016] FIG2 is a diagram showing an example structure of a video decoder to which an embodiment of the present application can be applied.

[0017] FIG3 is a schematic diagram of a coding and decoding framework using reference picture resampling (RPR) technology.

[0018] Figure 4 is a diagram showing an example of the NNSR structure.

[0019] Figure 5 is a schematic diagram of the residual network structure.

[0020] Figure 6 is a schematic diagram of the residual network structure of NNSR.

[0021] FIG7 is a schematic diagram of the residual network structure of a neural network based loop filter (NNLF).

[0022] Figure 8 is a schematic diagram of the residual network structure of NNPF.

[0023] FIG9 is a flowchart of a decoding method provided in an embodiment of the present application.

[0024] FIG10 is an example diagram of a residual image adjustment method provided in an embodiment of the present application.

[0025] FIG11 is a flow chart of the encoding method provided in an embodiment of the present application.

[0026] FIG12 is a schematic diagram of the encoding and decoding framework of the RPR technology provided in an embodiment of the present application.

[0027] FIG13 is a schematic diagram of the structure of a decoder provided in one embodiment of the present application.

[0028] FIG14 is a schematic diagram of the structure of a decoder provided in another embodiment of the present application.

[0029] FIG15 is a schematic diagram of the structure of an encoder provided in one embodiment of the present application.

[0030] FIG16 is a schematic diagram of the structure of an encoder provided in another embodiment of the present application. DETAILED DESCRIPTION

[0031] FIG1 is a schematic block diagram of a video encoder according to an embodiment of the present application.

[0032] It should be understood that the video encoder 100 can be used to perform lossy compression or lossless compression on an image. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0033] The video encoder 100 can be applied to image data in a luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0034] For example, the video encoder 100 reads video data, and for each image in the video data, divides the image into a number of coding tree units (CTUs). In some examples, CTUs may be referred to as "tree blocks", "largest coding units" (LCUs) or "coding tree blocks" (CTBs). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chroma sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding. A CU may be a rectangular block or a square block. A CU can be further divided into prediction units (PUs) and transform units (TUs), allowing for separation of coding, prediction, and transform, and greater flexibility in processing. In one example, a CTU is divided into CUs using a quadtree, and a CU is divided into TUs and PUs using a quadtree.

[0035] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0036] In some embodiments, as shown in FIG1 , the video encoder 100 may include a prediction unit 110, a residual unit 120, a transform / quantization unit 130, an inverse transform / quantization unit 140, a reconstruction unit 150, a loop filter unit 160, a decoded image buffer 170, and an entropy coding unit 180. It should be noted that the video encoder 100 may include more, fewer, or different functional components.

[0037] Optionally, in this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstruction block.

[0038] In some embodiments, the prediction unit 110 includes an inter-frame prediction unit 111 and an intra-frame prediction unit 112. Because there is a strong correlation between adjacent pixels in a video image, intra-frame prediction is used in video coding and decoding technologies to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent images in a video, inter-frame prediction is used in video coding and decoding technologies to eliminate temporal redundancy between adjacent images, thereby improving coding efficiency.

[0039] The inter-frame prediction unit 111 can be used for inter-frame prediction. Inter-frame prediction can include motion estimation and motion compensation. It can refer to image information from different images. Inter-frame prediction uses motion information to find a reference block from the reference image and generate a prediction block based on the reference block to eliminate temporal redundancy. Inter-frame prediction uses motion information to find a reference block from the reference image and generate a prediction block based on the reference block. Motion information includes the reference image list in which the reference image is located, the reference image index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference image. Here, the integer pixel or fractional pixel block in the reference image found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Reprocessing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.

[0040] The intra-frame prediction unit 112 only refers to information of the same image to predict pixel information within the current code image block to eliminate spatial redundancy.

[0041] Intra-frame prediction has multiple prediction modes. For example, the H-series international digital video coding standard H.264 / AVC has eight angular prediction modes and one non-angular prediction mode. H.265 / HEVC expands this to 33 angular prediction modes and two non-angular prediction modes. HEVC uses planar, DC, and 33 angular modes for a total of 35 intra-frame prediction modes. Versatile video coding (VVC) uses planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes.

[0042] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of high-definition and ultra-high-definition digital video development.

[0043] Residual unit 120 may generate a residual block for a CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, residual unit 120 may generate a residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel blocks of the CU and the corresponding sample in the prediction blocks of the PUs of the CU.

[0044] The transform / quantization unit 130 may quantize the transform coefficients. The transform / quantization unit 130 may quantize the transform coefficients associated with the TUs of the CU based on a quantization parameter (QP) value associated with the CU. The video encoder 100 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0045] The inverse transform / quantization unit 140 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0046] Reconstruction unit 150 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 110 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 100 can reconstruct the pixel blocks of the CU.

[0047] The loop filter unit 160 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent coded pixels. For example, it can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with the CU.

[0048] In some embodiments, the loop filtering unit 160 includes a deblocking filtering unit and a sample adaptive offset / adaptive loop filtering (SAO / ALF) unit, wherein the deblocking filtering unit is used to remove blocking effects, and the SAO / ALF unit is used to remove ringing effects.

[0049] The decoded image buffer 170 may store the reconstructed pixel blocks. The inter-prediction unit 111 may use a reference image containing the reconstructed pixel blocks to perform inter-prediction on PUs of other images. In addition, the intra-prediction unit 112 may use the reconstructed pixel blocks in the decoded image buffer 170 to perform intra-prediction on other PUs in the same image as the CU.

[0050] The entropy encoding unit 180 may receive the quantized transform coefficients from the transform / quantization unit 130. The entropy encoding unit 180 may perform one or more entropy encoding operations on the quantized transform coefficients to generate entropy-encoded data.

[0051] FIG2 is a schematic block diagram of a video decoder according to an embodiment of the present application.

[0052] 2 , video decoder 200 includes an entropy decoding unit 210, a prediction unit 220, an inverse quantization / transformation unit 230, a reconstruction unit 240, a loop filter unit 250, and a decoded picture buffer 260. It should be noted that video decoder 200 may include more, fewer, or different functional components.

[0053] Video decoder 200 may receive a bitstream. Entropy decoding unit 210 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, entropy decoding unit 210 may parse the entropy-encoded syntax elements in the bitstream. Prediction unit 220, inverse quantization / transform unit 230, reconstruction unit 240, and loop filter unit 250 may decode video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.

[0054] In some embodiments, the prediction unit 220 includes an intra-frame prediction unit 222 and an inter-frame prediction unit 221 .

[0055] The intra-frame prediction unit 222 may perform intra-frame prediction to generate a prediction block for the PU. The intra-frame prediction unit 222 may use an intra-frame prediction mode to generate a prediction block for the PU based on the pixel blocks of spatially neighboring PUs. The intra-frame prediction unit 222 may also determine the intra-frame prediction mode of the PU based on one or more syntax elements parsed from the codestream.

[0056] The inter-frame prediction unit 221 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. In addition, if a PU is encoded using inter-frame prediction, the entropy decoding unit 210 may parse the motion information of the PU. The inter-frame prediction unit 221 may determine one or more reference blocks of the PU based on the motion information of the PU. The inter-frame prediction unit 221 may generate a prediction block for the PU based on the one or more reference blocks of the PU.

[0057] The inverse quantization / transform unit 230 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 230 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0058] After inverse quantizing the transform coefficients, the inverse quantization / transform unit 230 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0059] Reconstruction unit 240 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, reconstruction unit 240 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0060] The loop filtering unit 250 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0061] The video decoder 200 may store the reconstructed image of the CU in the decoded image buffer 260. The video decoder 200 may use the reconstructed image in the decoded image buffer 260 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0062] The basic process of video encoding and decoding is as follows: At the encoder end, an image is divided into blocks. For the current block, the prediction unit 110 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 120 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block can also be referred to as residual information. This residual block undergoes transformation and quantization by the transform / quantization unit 130, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 130 can be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 130 can be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding unit 180 receives the quantized transform coefficients output by the transform and quantization unit 130, performs entropy coding on these quantized transform coefficients, and outputs a bitstream. For example, the entropy coding unit 180 can eliminate character redundancy based on the target context model and probability information of the binary bitstream.

[0063] At the decoding end, the entropy decoding unit 210 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 220 uses intra-frame prediction or inter-frame prediction on the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation unit 230 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 240 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 250 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference image for inter-frame prediction of subsequent images.

[0064] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and analyzes the existing information to determine the same block division information, prediction, transform, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical.

[0065] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0066] To meet the requirements of real-time video coding, some video codec standards (such as VVC) incorporate RPR technology, building upon the basic video coding framework. RPR adaptively adjusts image resolution within the bitstream based on network conditions, without requiring the insertion of instantaneous decoder refresh (IDR) frames or intra random access picture (IRAP) frames. When network bandwidth is low, RPR downsamples the original frame and encodes a low-resolution (LR) frame. When network bandwidth improves, RPR can encode a high-resolution (HR) original frame. The codec framework using RPR is shown in Figure 3. First, the input image is downsampled through preprocessing. The downsampled image is then encoded and decoded. Finally, the decoded reconstructed image is upsampled through postprocessing to obtain an image with the same resolution as the original input image, which serves as a reference image for subsequent frames to be encoded.

[0067] Traditional methods for post-processing (i.e., upsampling) in RPR include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. In recent years, with the development of deep learning technology, the exploration of NNSR has also gradually expanded.

[0068] After years of continuous research and optimization, a NNSR has been adopted as the baseline super-resolution tool for the neural network based video coding (NNVC) general reference software (NCS) due to its superior performance and complexity. The structure of this baseline NNSR is shown in Figure 4.

[0069] Referring to Figure 4, the baseline NNSR adopts the structure of a convolutional network, taking low-resolution reconstructed images and predicted images, as well as a variety of auxiliary information as network input. The various auxiliary information may include, for example, quantizer parameters (QP, or base QP) of general test conditions, slice-level QP (slice QP), etc. Then, after processing through multiple layers of convolution and residual blocks, the network outputs the predicted original resolution residual image. Next, the predicted original resolution residual image is added pixel by pixel to the original resolution reconstructed image obtained by upsampling based on the RPR technology to obtain the original resolution reconstructed image after upsampling by the neural network.

[0070] In the field of deep learning, the concept of residual learning has been proposed based on the classic residual network (ResNet) structure. Through a simple skip connection structure from the input to the output, the residual network structure allows the network to focus on learning the residual information of the image, thereby improving the network's learning ability and prediction performance. Its basic structure is shown in Figure 5.

[0071] The baseline NNSR also uses the classic residual network architecture, and its basic architecture is shown in Figure 6. Referring to Figure 6, input represents a variety of information input to the network, such as a low-resolution reconstructed image, a low-resolution predicted image, and a variety of auxiliary information. The various auxiliary information may include, for example, the QP (base QP) of the general test conditions, the slice-level QP (slice QP), etc. NNSR represents a neural network-based super-resolution device, which is mainly composed of residual blocks (ResBlock). NNSR goes through multiple layers of cascaded residual blocks and finally outputs the residual image Res of the original resolution predicted by the network. RPR represents the reconstructed image Rec of the original resolution obtained using the traditional upsampling method. "+" means that the residual image Res and the reconstructed image Rec are added pixel by pixel to obtain the upsampled reconstructed image Cnn of the original resolution output by the neural network.

[0072] As can be seen from the description above, NNSR uses a neural network to predict the residual information between the reconstructed image and the original image. Then, through a simple addition operation, NNSR superimposes this residual information on the reconstructed image to produce a super-resolved image. The quality of the super-resolved image output by NNSR is closer to that of the original image. This shows that neural networks based on the residual network architecture have the ability to predict residual information. In addition to NNSR, other neural network tools based on the residual network architecture include NNPF and NNLF, which also have the ability to predict residual information.

[0073] For example, as shown in Figure 7, Rec represents the reconstructed image output by the codec, and NNLF represents a neural network-based loop filter. NNLF is designed based on a residual network architecture, so the network prediction output is residual information. The NNLF is trained to predict the residual information between the input reconstructed image and the original image. Finally, through a simple addition operation, the residual information is superimposed on the input reconstructed image, resulting in a filtered image with a quality closer to the original image.

[0074] For example, as shown in Figure 8, Rec represents the reconstructed image output by the codec, and NNPF represents a neural network post-processing filter. Designed based on the residual network architecture, the network prediction output is residual information. The residual information output by the NNPF is added to the reconstructed image to obtain the final neural network reconstructed image, which serves as the filtered image obtained through post-processing.

[0075] Residual-based neural networks can be trained to predict image residual information. However, the network training process and the actual encoding and testing process are not completely consistent. For example, the training data set used for training and the test data set used for testing are not completely consistent. Furthermore, the configuration information of the training and testing environments is not completely consistent. This inconsistency can lead to errors in the neural network's output information, thereby reducing encoding and decoding performance.

[0076] In response to the above problems, an embodiment of the present application proposes an encoding method, including: determining a first reconstructed image; determining a first residual image corresponding to the first reconstructed image based on a neural network; adjusting the residual value in the first residual image to determine a second residual image; and determining a second reconstructed image based on the first reconstructed image and the second residual image.

[0077] An embodiment of the present application also provides a decoding method, including: parsing a code stream to determine a first reconstructed image; determining a first residual image corresponding to the first reconstructed image based on a neural network; adjusting the residual value in the first residual image to determine a second residual image; and determining a second reconstructed image based on the first reconstructed image and the second residual image.

[0078] As can be seen from the above description, the embodiment of the present application does not directly use the residual information determined by the neural network (corresponding to the first residual image described above), but instead adjusts the residual information output by the neural network before use. Adjusting the residual information may correct errors caused by inconsistencies between the training and actual testing processes, thereby improving codec performance.

[0079] The following describes the embodiments of the present application from the decoding end with reference to the accompanying drawings.

[0080] FIG9 is a flow chart of a decoding method according to an embodiment of the present application. The method of FIG9 can be applied to a decoder. The decoder can be, for example, a decoder that supports decoding based on a neural network tool (such as NNSR or NNPF).

[0081] 9 , in steps S910 to S920 , the code stream is parsed to determine a first reconstructed image; and a first residual image corresponding to the first reconstructed image is determined based on a neural network.

[0082] It should be noted that the image mentioned in the embodiment of the present application (such as a reconstructed image or a residual image) may refer to a frame of image, or may refer to an image block (such as a coding block) in a frame of image. In addition, the image mentioned in the embodiment of the present application may refer to an image corresponding to a color component, or may refer to an image corresponding to multiple color components. Taking the image including three color components Y, Cb, and Cr as an example, the image mentioned in the embodiment of the present application may refer to an image corresponding to the Y component, an image corresponding to the Cb component, or an image corresponding to the Cr component. Taking the image including luminance (luma) and chroma (chroma) as an example, the image mentioned in the embodiment of the present application may refer to an image corresponding to the luminance component, or may refer to an image corresponding to the chroma component.

[0083] The first residual image may include predicted residual information of the first reconstructed image output by the neural network. The neural network mentioned in the embodiment of the present application may be a neural network based on a residual network structure. Alternatively, the neural network mentioned in the embodiment of the present application may refer to a neural network capable of predicting residual information, as an example, the neural network may be NNSR or NNPF.

[0084] It should be understood that the implementation of steps S910 to S920 is related to the type of neural network, and the embodiment of the present application does not specifically limit this.

[0085] Taking the neural network as NNSR as an example, step S910 may include: parsing the code stream to determine the third reconstructed image; upsampling the third reconstructed image to determine the first reconstructed image. The third reconstructed image mentioned here may be a low-resolution reconstructed image. The low-resolution reconstructed image can be obtained through traditional video decoding. For example, the code stream can be parsed to determine the quantization coefficient; the quantization coefficient is inversely quantized to determine the transformation coefficient; the transformation coefficient is inversely transformed to determine the residual value; and the third reconstructed image is determined based on the predicted value and the residual value. There are many ways to upsample the third reconstructed image. For example, the third reconstructed image can be upsampled based on the RPR technology. Exemplarily, the third reconstructed image can be upsampled using one or more of the following methods: nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. The resolution of the upsampled third reconstructed image can be called the original resolution reconstructed image.

[0086] Furthermore, in some implementations, step S920 may include: inputting the third reconstructed image into the neural network to determine the first residual image, where the resolution of the first residual image is higher than the resolution of the third reconstructed image. In other words, inputting the low-resolution reconstructed image into the neural network to determine the residual image of the original resolution. It should be understood that in addition to the third reconstructed image, other information may be input into the neural network so that the neural network can better predict the first residual image. For example, the first predicted image may be input into the neural network. The first predicted image may be a predicted image with the same resolution as the third reconstructed image, and therefore, the first predicted image may also be referred to as a low-resolution predicted image. For another example, other auxiliary information may also be input into the neural network. The auxiliary information mentioned here may, for example, include one or more of the following: QP of general test conditions, slice-level QP, etc.

[0087] Taking the neural network as an NNPF as an example, the first reconstructed image in step S910 can be obtained based on a traditional video decoding process. For example, step S910 may include: parsing the bitstream to determine the quantization coefficients; performing inverse quantization on the quantization coefficients to determine the transform coefficients; performing inverse transformation on the transform coefficients to determine the residual values; and determining the first reconstructed image based on the predicted values ​​and the residual values. Step S920 may include: inputting the first reconstructed image into the neural network to determine the first residual image. In other words, the first reconstructed image can be input into the neural network so that the neural network predicts the residual information of the first reconstructed image, thereby obtaining the first residual image.

[0088] Continuing to refer to Figure 9, in step S930, the residual value in the first residual image is adjusted to determine the second residual image. The embodiment of the present application does not specifically limit the adjustment method of the residual value. The residual value in the first residual image can be increased, or the residual value in the first residual image can be decreased. A certain residual adjustment formula can be used to adjust the residual value in the first residual image to obtain a second residual image. Alternatively, one or more fixed offset values ​​(which can be positive numbers) can be used to offset the residual value in the first residual image (i.e., residual offset adjustment (ROA)) to obtain a second residual image.

[0089] After testing, it was found that in most cases, the residual value in the first residual image is larger than the actual residual value. Therefore, the residual value in the first residual image can be adjusted with the goal of reducing the residual value of the first residual image.

[0090] As an example, assuming that the first residual image includes a first residual value, and the second residual image includes a second residual value corresponding to the first residual value (i.e., the second residual value is the residual value obtained after adjusting the first residual value), taking the use of a fixed first bias value to adjust the residual value of the first residual image as an example, since the residual values ​​in a residual image may be positive or negative, the adjustment of the first bias value can make the first residual value and the second residual value satisfy at least one of the following: if the first residual value is positive, the second residual value is equal to the difference between the first residual value and the first bias value; if the first residual value is negative, the second residual value is equal to the sum of the first residual value and the first bias value; if the first residual value is 0, the second residual value is equal to the first residual value. After the above adjustment, the overall absolute residual value of the first residual image will become smaller. Taking Figure 10 as an example, assuming that the first bias value is 1, Figure 10 (a) represents the residual image output by the original NNSR, and Figure 10 (b) represents the residual image after adjustment. Comparing the residual values ​​of the two residual images in Figure 10 shows that the positive residual values ​​in the residual image output by the original NNSR are all decremented by 1, the negative residual values ​​are all incremented by 1, and the residual value of 0 remains unchanged. This adjustment reduces the overall absolute residual value of the residual image output by the original NNSR. Using a fixed offset value to adjust the residual value is very simple to implement and does not increase decoding complexity.

[0091] It should be understood that the above is explained by taking the first bias value as an example of a bias value. In fact, the first bias value mentioned in the embodiment of the present application may refer to a bias value, or it may be a group of bias values ​​formed by multiple bias values ​​with different values. The multiple bias values ​​can be used to adjust the residual values ​​within multiple value ranges respectively. The multiple bias values ​​can be understood as multiple bias values ​​of different precision. For larger residual values, a bias value with lower precision (larger value) can be used to adjust it; for smaller residual values, a bias value with higher precision (smaller value) can be used to adjust it.

[0092] As an example, assuming that for the current pixel point in the current frame, the corresponding residual value is res, and the offset value corresponding to res is ROA_FACTOR, the value of ROA_FACTOR can be derived using the following strategy (expressed in pseudo code):

[0093] if (res=0) RO_FACTOR=0;

[0094] else if(0 <res<=x1) RO_FACTOR=a1;

[0095] else if(x1 <res<=x2) RO_FACTOR=a2;

[0096] else if(x2 <res) RO_FACTOR=a3;

[0097] else if(y1<=res<0) RO_FACTOR=b1;

[0098] else if(y2<=res <y1) RO_FACTOR=b2;

[0099] else if(res <y2) RO_FACTOR=b3;

[0100] Among them, {x1, x2, x3} represents positive residual values, {y1, y2, y3} represents negative residual values, and {a1, a2, a3} and {b1, b2, b3} are preset candidate bias values.

[0101] Continuing with FIG9 , in step S940, a second reconstructed image is determined based on the first reconstructed image and the second residual image. For example, the first reconstructed image and the second residual image may be added pixel by pixel to obtain the second reconstructed image. For example, if the neural network is NNSR, the second reconstructed image may be a neural network upsampled reconstructed image of the original resolution. For example, if the neural network is NNPF, the second reconstructed image may be a filtered image obtained through post-processing.

[0102] As can be seen from the above description, the embodiment of the present application does not directly use the residual information determined by the neural network (corresponding to the first residual image mentioned above), but adjusts the residual information output by the neural network before use (i.e., using the second residual image). The adjustment of the residual information may correct errors caused by the inconsistency between the training and actual testing processes, thereby improving decoding performance.

[0103] The above describes in detail how to adjust the residual image determined based on the neural network. The adjustment of the residual image by the decoding end can be based on auxiliary information in the code stream (such as high-level syntax elements). The following is a detailed example of the auxiliary information that may be used to implement the residual image adjustment scheme at the decoding end. It should be understood that the auxiliary information mentioned below is optional information. In some implementations, the codec can adjust the residual image according to the same predefined rules without the need for such auxiliary information.

[0104] In some implementations, the code stream may carry first identification information. Therefore, during the decoding process, the code stream may be parsed to determine the first identification information. The first identification information may be used to indicate whether residual adjustment is performed on the first residual image (or whether residual adjustment is allowed on the first residual image). The first identification information may include a first value and a second value. The first value may be, for example, 1 or true. The second value may be, for example, 0 or false. If the value of the first identification information is the first value, it instructs the decoding end to perform residual adjustment on the first residual image; if the value of the first identification information is the second value, it may instruct the decoding end not to perform residual adjustment on the first residual image.

[0105] The above-mentioned first identification information can be a flag bit at the picture level. Therefore, the first identification information can be called a picture-level residual information adjustment enable flag. The first identification information can be represented by picture_roa_enable_flag[comp], where comp represents the current color component. For example, during the decoding process of the current color component, the picture-level residual information adjustment enable flag picture_roa_enable_flag[comp] can be parsed. If picture_roa_enable_flag[comp] is "1", residual adjustment is performed on the current color component; if picture_roa_enable_flag[comp] is "0", residual adjustment is not performed on the current color component.

[0106] In some implementations, the codestream may carry second identification information. Therefore, during the decoding process, the codestream can be parsed to determine the second identification information. The second identification information is used to indicate whether residual adjustment is performed on the image sequence (or whether residual adjustment is allowed on the image sequence). The second identification information includes a third value and a fourth value. The third value can be, for example, 1 or true. The fourth value can be, for example, 0 or false. The third value indicates that residual adjustment is performed on the image sequence, and the fourth value indicates that residual adjustment is not performed on the image sequence. The second identification information can be represented by the roa_enable_flag. It should be understood that the second identification information and the first identification information mentioned above can be used in combination. For example, when the value of the second identification information is the third value (i.e., residual adjustment is allowed on the image sequence), the value of the first identification information can be determined. If the value of the first identification information is the first value (i.e., residual adjustment is allowed on the first residual image), the first residual image is adjusted; if the value of the first identification information is the second value (i.e., residual adjustment is not allowed on the first residual image), the first residual image is not adjusted. For another example, when the value of the second identification information is the fourth value (i.e., residual adjustment is not allowed for the image sequence), residual adjustment is not performed on the images in the current sequence. In this case, the decoder does not need to parse the first identification information and defaults to the second value for the first identification information of each frame in the current sequence.

[0107] The residual adjustment method used by the codec may include only one residual adjustment method or multiple residual adjustment methods. If only one residual adjustment method is included, when the decoder determines to adjust the first residual image, the residual adjustment method may be directly used for adjustment. If multiple residual adjustment methods are included, the code stream may carry index information indicating the currently used residual adjustment method (the index information may be carried in supplemental enhancement information (SEI)). Still taking the residual adjustment of the first residual image as an example, the code stream may be parsed to determine the first index information. The first index information may be used to indicate the residual adjustment method of the first residual image. The first index information may indicate which of the preset multiple residual adjustment methods the residual adjustment method of the first residual image is. For example, the preset residual adjustment method includes two residual adjustment methods. The index of the first residual adjustment method is 0, and the index of the second residual adjustment method is 1. If the code stream is parsed and it is determined that the value of the first index information is 1, it may be determined that the second residual adjustment method is used to adjust the residual value of the first residual image.

[0108] As mentioned above, the residual value can be adjusted based on the bias value. Therefore, in some implementations, different candidate bias values ​​can represent different residual adjustment methods. For example, two candidate bias values ​​can be pre-set: 1 and 2. Among them, the value of the first index information corresponding to the candidate bias value 1 is 0, and the value of the first index information corresponding to the candidate bias value 2 is 1. If the code stream is parsed and it is determined that the value of the first index information is 1, it can be determined that the bias value 2 is used to adjust the residual value of the first residual image. For example, if the first residual value in the first residual image is a positive number, the first residual value is subtracted by 2; if the first residual value in the first residual image is a negative number, the first residual value is added by 2; if the first residual value in the first residual image is 0, the first residual value remains unchanged.

[0109] It should also be understood that in some implementations, the first index information may be used in combination with the first identification information mentioned above. For example, if the first identification information indicates that residual adjustment should be performed on the first residual image, the first index information may be parsed to determine the residual adjustment method. For another example, if the first identification information indicates that residual adjustment should not be performed on the first residual image, the first index information may not be parsed or parsing of the first index information may be skipped.

[0110] The above description, in conjunction with Figure 9 , details the decoding method provided by the embodiment of the present application from the perspective of a decoder. The following description, in conjunction with Figure 11 , details the encoding method provided by the embodiment of the present application from the perspective of an encoder.

[0111] Figure 11 is a flow chart of an encoding method provided in an embodiment of the present application. The method of Figure 11 can be applied to an encoder. The encoder can, for example, be an encoder that supports encoding based on a neural network tool (such as NNSR or NNPF).

[0112] 11 , in steps S1110 to S1120 , a first reconstructed image is determined; and a first residual image corresponding to the first reconstructed image is determined based on a neural network.

[0113] It should be noted that the image mentioned in the embodiment of the present application (such as a reconstructed image or a residual image) may refer to a frame of image, or may refer to an image block (such as a coding block) in a frame of image. In addition, the image mentioned in the embodiment of the present application may refer to an image corresponding to a color component, or may refer to an image corresponding to multiple color components. Taking the image including three color components Y, Cb, and Cr as an example, the image mentioned in the embodiment of the present application may refer to an image corresponding to the Y component, an image corresponding to the Cb component, or an image corresponding to the Cr component. Taking the image including luminance (luma) and chroma (chroma) as an example, the image mentioned in the embodiment of the present application may refer to an image corresponding to the luminance component, or may refer to an image corresponding to the chroma component.

[0114] The first residual image may include predicted residual information of the first reconstructed image output by the neural network. The neural network mentioned in the embodiment of the present application may be a neural network based on a residual network structure. Alternatively, the neural network mentioned in the embodiment of the present application may refer to a neural network capable of predicting residual information, as an example, the neural network may be NNSR or NNPF.

[0115] It should be understood that the implementation of steps S1110 to S1120 is related to the type of neural network, and the embodiments of the present application do not specifically limit this.

[0116] Taking the neural network as NNSR as an example, step S1110 may include: upsampling the third reconstructed image to determine the first reconstructed image. The third reconstructed image mentioned here may be a low-resolution reconstructed image. The low-resolution reconstructed image can be obtained through traditional video coding methods. For example, the image to be encoded may first be predicted, transformed, quantized, and other operations to obtain quantization coefficients. Then, the quantization coefficients may be inversely quantized, inversely transformed, and other operations to determine the third reconstructed image. There may be multiple ways to upsample the third reconstructed image. For example, the third reconstructed image may be upsampled based on RPR technology. Exemplarily, the third reconstructed image may be upsampled using one or more of the following methods: nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. The resolution of the upsampled third reconstructed image may be referred to as the original resolution reconstructed image.

[0117] Furthermore, in some implementations, step S1120 may include: inputting the third reconstructed image into the neural network to determine the first residual image, where the resolution of the first residual image is higher than the resolution of the third reconstructed image. In other words, inputting the low-resolution reconstructed image into the neural network to determine the residual image of the original resolution. It should be understood that in addition to the third reconstructed image, other information may be input into the neural network so that the neural network can better predict the first residual image. For example, the first predicted image may be input into the neural network. The first predicted image may be a predicted image with the same resolution as the third reconstructed image, and therefore, the first predicted image may also be referred to as a low-resolution predicted image. For another example, other auxiliary information may also be input into the neural network. The auxiliary information mentioned here may, for example, include one or more of the following: QP of general test conditions, slice-level QP, etc.

[0118] Taking the neural network as an example, the first reconstructed image in step S1110 can be obtained based on a traditional video encoding process. For example, step S1110 may include performing operations such as prediction, transformation, and quantization on the image to be encoded to determine the first reconstructed image. Step S1120 may include inputting the first reconstructed image into the neural network to determine the first residual image. In other words, the first reconstructed image can be input into the neural network, causing the neural network to predict residual information of the first reconstructed image, thereby obtaining the first residual image.

[0119] In step S1130, the residual value in the first residual image is adjusted to determine the second residual image. The embodiment of the present application does not specifically limit the adjustment method of the residual value. The residual value in the first residual image can be increased, or the residual value in the first residual image can be decreased. A certain residual adjustment formula can be used to adjust the residual value in the first residual image to obtain the second residual image. Alternatively, one or more fixed offset values ​​(which can be positive numbers) can be used to offset the residual value in the first residual image (i.e., residual offset adjustment (ROA)) to obtain the second residual image.

[0120] After testing, it was found that in most cases, the residual value in the first residual image is larger than the actual residual value. Therefore, the residual value in the first residual image can be adjusted with the goal of reducing the residual value of the first residual image.

[0121] As an example, assuming that the first residual image includes a first residual value, and the second residual image includes a second residual value corresponding to the first residual value (i.e., the second residual value is the residual value obtained after adjusting the first residual value), taking the use of a fixed first bias value to adjust the residual value of the first residual image as an example, since the residual values ​​in a residual image may be positive or negative, the adjustment of the first bias value can make the first residual value and the second residual value satisfy at least one of the following: if the first residual value is positive, the second residual value is equal to the difference between the first residual value and the first bias value; if the first residual value is negative, the second residual value is equal to the sum of the first residual value and the first bias value; if the first residual value is 0, the second residual value is equal to the first residual value. After the above adjustment, the overall absolute residual value of the first residual image will become smaller. Taking Figure 10 as an example, assuming that the first bias value is 1, Figure 10 (a) represents the residual image output by the original NNSR, and Figure 10 (b) represents the residual image after adjustment. Comparing the residual values ​​of the two residual images in Figure 10 shows that the positive residual values ​​in the residual image output by the original NNSR are all decremented by 1, the negative residual values ​​are all incremented by 1, and the residual value of 0 remains unchanged. This adjustment reduces the overall absolute residual value of the residual image output by the original NNSR. Using a fixed offset value to adjust the residual value is very simple to implement and barely increases coding complexity.

[0122] It should be understood that the above is explained by taking the first bias value as an example of a bias value. In fact, the first bias value mentioned in the embodiment of the present application may refer to a bias value, or it may be a group of bias values ​​formed by multiple bias values ​​with different values. The multiple bias values ​​can be used to adjust the residual values ​​within multiple value ranges respectively. The multiple bias values ​​can be understood as multiple bias values ​​of different precision. For larger residual values, a bias value with lower precision (larger value) can be used to adjust it; for smaller residual values, a bias value with higher precision (smaller value) can be used to adjust it.

[0123] As an example, assuming that for the current pixel point in the current frame, the corresponding residual value is res, and the offset value corresponding to res is ROA_FACTOR, the value of ROA_FACTOR can be derived using the following strategy (expressed in pseudo code):

[0124] if (res=0) RO_FACTOR=0;

[0125] else if(0 <res<=x1) RO_FACTOR=a1;

[0126] else if(x1 <res<=x2) RO_FACTOR=a2;

[0127] else if(x2 <res) RO_FACTOR=a3;

[0128] else if(y1<=res<0) RO_FACTOR=b1;

[0129] else if(y2<=res <y1) RO_FACTOR=b2;

[0130] else if(res <y2) RO_FACTOR=b3;

[0131] Among them, {x1, x2, x3} represents positive residual values, {y1, y2, y3} represents negative residual values, and {a1, a2, a3} and {b1, b2, b3} are preset candidate bias values.

[0132] Continuing with FIG. 11 , in step S1140, a second reconstructed image is determined based on the first reconstructed image and the second residual image. For example, the first reconstructed image and the second residual image may be added pixel by pixel to obtain the second reconstructed image. For example, if the neural network is NNSR, the second reconstructed image may be a neural network upsampled reconstructed image of the original resolution. For example, if the neural network is NNPF, the second reconstructed image may be a filtered image obtained through post-processing.

[0133] As can be seen from the above description, the embodiment of the present application does not directly use the residual information determined by the neural network (corresponding to the first residual image mentioned above), but adjusts the residual information output by the neural network before using it (i.e., using the second residual image). The adjustment of the residual information may correct errors caused by the inconsistency between the training and actual testing processes, thereby improving the encoding performance.

[0134] The above describes in detail how to adjust the residual image determined based on the neural network. The following describes in detail the relevant parameters (such as high-level syntax elements) that may be involved in the residual image adjustment process.

[0135] In some implementations, first identification information may be written into the codestream. The first identification information may be used to indicate whether residual adjustment is performed on the first residual image (or whether residual adjustment is allowed on the first residual image). The first identification information may include a first value and a second value. The first value may be, for example, 1 or true. The second value may be, for example, 0 or false. If the value of the first identification information is the first value, it indicates that residual adjustment is performed on the first residual image; if the value of the first identification information is the second value, it may indicate that residual adjustment is not performed on the first residual image.

[0136] The above-mentioned first identification information can be a flag bit at the picture level. Therefore, the first identification information can be called a picture-level residual information adjustment enable flag. The first identification information can be represented by picture_roa_enable_flag[comp], where comp represents the current color component. For example, during the encoding process of the current color component, the picture-level residual information adjustment enable flag picture_roa_enable_flag[comp] can be encoded. If picture_roa_enable_flag[comp] is "1", it indicates that residual adjustment is performed on the current color component; if picture_roa_enable_flag[comp] is "0", it indicates that residual adjustment is not performed on the current color component.

[0137] Whether to write the first identification information into the codestream can be determined based on the rate-distortion cost corresponding to the first reconstructed image and the second reconstructed image. The rate-distortion cost corresponding to the first reconstructed image can be determined by comparing the first reconstructed image with the original image. Similarly, the rate-distortion cost corresponding to the second reconstructed image can be determined by comparing the second reconstructed image with the original image. For example, if the rate-distortion cost corresponding to the second reconstructed image is less than the rate-distortion cost corresponding to the first reconstructed image, the value of the first identification information is the first value. For another example, if the rate-distortion cost corresponding to the second reconstructed image is greater than or equal to the rate-distortion cost corresponding to the first reconstructed image, the value of the first identification information is the second value.

[0138] In some implementations, the second identification information can be written into the code stream. The second identification information is used to indicate whether residual adjustment is performed on the image sequence (or whether residual adjustment is allowed on the image sequence). The second identification information includes a third value and a fourth value. The third value can be, for example, 1 or true. The fourth value can be, for example, 0 or false. The third value is used to indicate that residual adjustment is performed on the image sequence, and the fourth value is used to indicate that residual adjustment is not performed on the image sequence. The second identification information can be represented by roa_enable_flag. It should be understood that the second identification information and the first identification information mentioned above can be used in combination. For example, when the value of the second identification information is the third value (i.e., residual adjustment is allowed on the image sequence), the first identification information can continue to be encoded. For another example, when the value of the second identification information is the fourth value (i.e., residual adjustment is not allowed on the image sequence), the first identification information is not encoded.

[0139] The residual adjustment method used by the codec may include only one residual adjustment method or multiple residual adjustment methods. If only one residual adjustment method is included, when the encoder determines to adjust the first residual image, the residual adjustment method can be directly used for adjustment. If multiple residual adjustment methods are included, the encoder can write index information indicating the currently used residual adjustment method into the bitstream (the index information can be carried in SEI). Still taking the residual adjustment of the first residual image as an example, the first index information can be written into the bitstream. The first index information can be used to indicate that the first residual image uses a target residual adjustment method among multiple residual adjustment methods. For example, the preset residual adjustment method includes two residual adjustment methods. The index of the first residual adjustment method is 0, and the index of the second residual adjustment method is 1. If the value of the first index information is 1, it means that the second residual adjustment method is used to adjust the residual value of the first residual image.

[0140] As mentioned above, the residual value can be adjusted based on the bias value. Therefore, in some implementations, different candidate bias values ​​can represent different residual adjustment methods. For example, two candidate bias values ​​can be pre-set: 1 and 2. Among them, the value of the first index information corresponding to the candidate bias value 1 is 0, and the value of the first index information corresponding to the candidate bias value 2 is 1. If the value of the first index information is 1, it means that the residual value of the first residual image is adjusted using the bias value 2. For example, if the first residual value in the first residual image is a positive number, the first residual value is subtracted by 2; if the first residual value in the first residual image is a negative number, the first residual value is added by 2; if the first residual value in the first residual image is 0, the first residual value remains unchanged.

[0141] The target residual adjustment method ultimately used can be determined based on the rate-distortion costs corresponding to the multiple residual adjustment methods. For example, multiple residual images can be determined based on the multiple residual adjustment methods. Then, a target residual adjustment method is determined based on the rate-distortion costs corresponding to the multiple residual images (for example, the target residual adjustment method is the residual adjustment method with the optimal rate-distortion cost). For example, multiple reconstructed images can be first determined based on the multiple residual images, and then the rate-distortion costs corresponding to the multiple residual images can be determined by comparing the multiple reconstructed images with the original images.

[0142] In some implementations, the first index information may be used in combination with the first identification information mentioned above. For example, if the first identification information indicates that residual adjustment should be performed on the first residual image, the first index information may be parsed to determine the residual adjustment method. For another example, if the first identification information indicates that residual adjustment should not be performed on the first residual image, the first index information may not be parsed or parsing of the first index information may be skipped.

[0143] The following examples are used to describe the embodiments of the present application in more detail. It should be noted that the following examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. It is apparent that those skilled in the art can make various equivalent modifications or changes based on the examples given, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0144] This example proposes an image-level output information adjustment method to further optimize the encoding and decoding performance of NNSR by correcting the residual image output by NNSR.

[0145] The position of the adjustment module for outputting residual information in the codec is shown in Figure 12. The NNSR in Figure 12 is a certain NNSR in the related art, and ROA is the residual information adjustment module proposed in this example.

[0146] ROA is used to adjust the residual of the NNSR output. On the encoder side, the rate-distortion cost between the original NNSR output and the original image is obtained, as well as the rate-distortion cost between the residual-adjusted NNSR output and the original image. By comparing these two rate-distortion costs, the NNSR output is determined to be used. The selected output is encoded into the bitstream for the decoder to read. On the decoder side, after parsing the output actually used by the encoder, super-resolution restoration is performed on the reconstructed image.

[0147] This example is implemented in an RPR-based codec, specifically in the upsampling module.

[0148] The specific implementation of this example in the upsampling module on the encoder side is as follows:

[0149] Step a) determines whether the current sequence can use ROA based on the sequence-level flag roa_enable_flag. If roa_enable_flag is "1," the current sequence uses ROA to attempt residual information adjustment for the current sequence, and the process proceeds to step b). If roa_enable_flag is "0," the current sequence does not use ROA, and the process ends.

[0150] In step b), for the current color component of the current frame in the current sequence, use NNSR to upsample the low-resolution reconstructed image to predict a residual image at the original resolution. This predicted residual image at the original resolution is added pixel by pixel to the reconstructed image at the original resolution obtained by upsampling with RPR to obtain the neural network upsampled reconstructed image at the original resolution. Go to step c).

[0151] Step c) Adjust the original resolution residual image obtained in step b) and add it pixel by pixel to the original resolution reconstructed image obtained by RPR upsampling to obtain the original resolution reconstructed corrected image. Go to step d);

[0152] Step d) Compare the neural network upsampled reconstructed image and the original resolution reconstructed corrected image generated in this example with the original image and calculate the rate distortion cost, which is called Distortion (D). Compare the two costs. If D ROA <D NNSR , then the original resolution reconstructed and corrected image output by ROA is used as the final neural network up-sampled reconstructed image; otherwise, if D ROA ≥D NNSR , the neural network upsampled reconstructed image output by the original NNSR is still used as the final upsampled reconstructed image. Jump to step e).

[0153] Step e) Encode the picture-level residual information adjustment enable flag picture_roa_enable_flag[comp] of the current color component of the current frame of the current sequence into the SEI. If picture_roa_enable_flag[comp] is "1", further encode the index of the residual adjustment offset picture_roa_offset_Id[comp]. Jump to step f).

[0154] Step f): If the current frame has been processed, the next frame is loaded for processing and the process goes to step b).

[0155] The specific process of this example in the upsampling module on the decoding side is as follows:

[0156] In step a), the sequence-level flag roa_enable_flag is first parsed to determine whether ROA can be used in the current sequence. If roa_enable_flag is "1", ROA is used in the current sequence to attempt residual information adjustment for the current sequence, and the process jumps to step b). If roa_enable_flag is "0", ROA is not used in the current sequence, and the process ends.

[0157] In step b), for the current color component of the current frame in the current sequence, use NNSR to upsample the low-resolution reconstructed image to obtain a predicted original-resolution residual image. This residual image is then added pixel by pixel to the original-resolution reconstructed image obtained by upsampling with RPR to obtain the neural network upsampled reconstructed image at the original resolution. Go to step c).

[0158] Step c) For the current color component of the current frame of the current sequence, parse the image-level residual information adjustment enable flag picture_roa_enable_flag[comp]. If picture_roa_enable_flag[comp] is "1", further parse the index of the residual adjustment offset picture_roa_offset_Id[comp] and jump to step d); if picture_roa_enable_flag[comp] is "0", do not use ROA, and still use the original NNSR output neural network upsampled image as the final upsampled reconstructed image, and jump to step e);

[0159] In step d), the residual image is adjusted based on the index picture_roa_offset_Id[comp] of the resolved residual adjustment offset, and pixel-by-pixel addition is performed on the original resolution reconstructed image obtained by RPR upsampling to obtain the original resolution reconstructed corrected image. This original resolution reconstructed corrected image is used as the final upsampled reconstructed image, and the process proceeds to step e).

[0160] Step e): If the current frame has been processed, the next frame is loaded for processing and the process goes to step b).

[0161] The following is the pseudo code of the codec residual adjustment process:

[0162] The above code defines the following semantics:

[0163] Sequence level residual information adjustment enable flag roa_enable_flag;

[0164] Picture level residual information adjustment enable flag picture_roa_enable_flag[N];

[0165] When the picture-level residual information adjustment enable flag picture_roa_enable_flag[N] is 1, the following semantics are further defined:

[0166] The image-level residual adjustment offset index picture_roa_offset_Id[N] represents the serial number of the encoded residual adjustment offset; N can be 3, representing the three color components Y, Cb, and Cr; N can also be 2, representing the two color channels of brightness and chrominance.

[0167] The following describes the specific method of residual adjustment used in this example.

[0168] The basic purpose of residual adjustment is to reduce the output residual. Specifically, for a residual image, the residual value at each pixel can be positive or negative. In this example, according to the goal of reducing the residual, a fixed value (positive number) is subtracted from the positive residual value, and the negative residual value is added to the fixed value. When the residual value is 0, no adjustment is made, so that the overall absolute residual value becomes smaller. As shown in Figure 10, assuming that the fixed value of the adjustment bias is (+1), Figure 10 (a) represents the residual image output by the original NNSR, and Figure 10 (b) represents the adjusted residual image. Finally, the residual image is added pixel by pixel to the reconstructed image obtained by RPR upsampling to obtain the neural network upsampling reconstructed adjusted image of the original resolution.

[0169] This example sets multiple candidate offset values ​​for residual information adjustment. By calculating the rate-distortion cost, one of these offset values ​​is selected and its index is encoded into the bitstream for the decoder to read and process.

[0170] The following describes the test method and test results of this example.

[0171] Based on the baseline NNSR tool based on NNVC, the technical solution provided in this example was implemented for all frame types (including I, P, and B), and the performance was tested. In the specific simulation experiment, the candidate bias values ​​were set to 1 and 2.

[0172] Under the general test conditions of random access and all intra, the Joint Video Experts Team (JVET) standard sequences were tested and compared with the performance of the baseline NNSR. The results are shown in Tables 1 and 2. It should be noted that for the noodle-stirring tool, JVET only requires testing at 4K resolutions (Class A1 and Class A2). This example also tests 1080P sequences (Class B).

[0173] Table 1 Random Access Performance

[0174] Table 2 All Intra performance

[0175] Analyzing the data in the table above, we can see that by introducing residual adjustment, encoding performance can be further improved over current baseline super-resolution tools, with particularly strong performance on chrominance components. Furthermore, this solution only performs simple adjustments to the output of the super-resolution network, without increasing encoding and decoding complexity.

[0176] This solution proposes an image-level output information adjustment method to further improve the encoding and decoding performance of the neural network super-resolution device by correcting the output residual image. Because this adjustment method is simple to implement, it barely increases the encoding and decoding complexity.

[0177] The method embodiment of the present application is described in detail above in conjunction with Figures 1 to 12. The device embodiment of the present application is described in detail below in conjunction with Figures 13 to 16. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for parts not described in detail, reference can be made to the above method embodiment.

[0178] FIG13 is a schematic diagram of the structure of a decoder provided by one embodiment of the present application. The decoder 1300 shown in FIG13 includes a first determination unit 1310, a second determination unit 1320, a third determination unit 1330, and a fourth determination unit 1340. The first determination unit 1310 is configured to parse the code stream and determine a first reconstructed image. The second determination unit 1320 is configured to determine a first residual image corresponding to the first reconstructed image based on a neural network. The third determination unit 1330 is configured to adjust the residual value in the first residual image to determine a second residual image. The fourth determination unit 1340 is configured to determine a second reconstructed image based on the first reconstructed image and the second residual image.

[0179] In some implementations, the decoder 1300 further includes: a fifth determining unit configured to parse the code stream and determine first identification information, where the first identification information is used to indicate whether to perform residual adjustment on the first residual image.

[0180] In some implementations, the first identification information includes a first value and a second value, the first value is used to indicate that residual adjustment is performed on the first residual image, and the second value is used to indicate that residual adjustment is not performed on the first residual image.

[0181] In some implementations, the decoder 1300 further includes: a sixth determining unit configured to parse the code stream and determine second identification information, where the second identification information is used to indicate whether to perform residual adjustment on the image sequence.

[0182] In some implementations, the second identification information includes a third value and a fourth value, the third value is used to indicate that residual adjustment is performed on the image sequence, and the fourth value is used to indicate that residual adjustment is not performed on the image sequence.

[0183] In some implementations, the decoder 1300 further includes: a seventh determining unit configured to parse the code stream and determine first index information, where the first index information is used to indicate a residual adjustment method of the first residual image.

[0184] In some implementations, the first index information is used to indicate a first offset value, and the residual value of the second residual image is determined based on the first offset value and the residual value of the first residual image.

[0185] In some implementations, the first index information belongs to supplemental enhancement information.

[0186] In some implementations, the first residual image includes a first residual value, the second residual image includes a second residual value corresponding to the first residual value, and the first residual value and the second residual value satisfy at least one of the following:

[0187] If the first residual value is a positive number, the second residual value is equal to the difference between the first residual value and the first offset value;

[0188] If the first residual value is a negative number, the second residual value is equal to the sum of the first residual value and the first offset value;

[0189] If the first residual value is 0, the second residual value is equal to the first residual value.

[0190] In some implementations, the first bias value includes a plurality of bias values ​​having different values, and the plurality of bias values ​​are respectively used to adjust residual values ​​within a plurality of value ranges.

[0191] In some implementations, the neural network is a neural network-based super-resolution analyzer.

[0192] In some implementations, the first determining unit 1310 is configured to: parse the code stream to determine the third reconstructed image; and upsample the third reconstructed image to determine the first reconstructed image.

[0193] In some implementations, the second determining unit 1320 is configured to: input the third reconstructed image into the neural network to determine the first residual image, where the resolution of the first residual image is higher than the resolution of the third reconstructed image.

[0194] In some implementations, the neural network is a neural network-based post-filter.

[0195] In some implementations, the second determining unit 1320 is configured to: input the first reconstructed image into the neural network to determine the first residual image.

[0196] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0197] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0198] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the decoder 1300. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the decoding method in the first embodiment.

[0199] Based on the composition of the above-mentioned decoder and the computer-readable storage medium, refer to Figure 14, which shows a specific hardware structure diagram of the decoder provided by an embodiment of the present application. As shown in Figure 14, the decoder 1400 may include: a communication interface 1410, a memory 1420 and a processor 1430; each component is coupled together through a bus system 1440. It can be understood that the bus system 1440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 1440 in Figure 14. Among them,

[0200] The communication interface 1410 is used to receive and send signals when sending and receiving information with other external network elements;

[0201] Memory 1420, for storing computer programs;

[0202] The processor 1430 is configured to, when running the computer program, execute:

[0203] Parsing the code stream to determine the first reconstructed image;

[0204] determining a first residual image corresponding to the first reconstructed image according to a neural network;

[0205] adjusting the residual value in the first residual image to determine a second residual image;

[0206] A second reconstructed image is determined according to the first reconstructed image and the second residual image.

[0207] It is understood that the memory 1420 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1420 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0208] The processor 1430 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1430. The above-mentioned processor 1430 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1420 , and the processor 1430 reads the information in the memory 1420 and completes the steps of the above method in combination with its hardware.

[0209] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0210] Optionally, as another embodiment, the processor 1430 is further configured to execute the decoding method described in the above embodiment when running the computer program.

[0211] FIG15 is a schematic diagram of the structure of an encoder provided by one embodiment of the present application. The decoder 1500 shown in FIG15 includes a first determination unit 1510, a second determination unit 1520, a third determination unit 1530, and a fourth determination unit 1540. The first determination unit 1510 is configured to determine a first reconstructed image. The second determination unit 1520 is configured to determine a first residual image corresponding to the first reconstructed image based on a neural network. The third determination unit 1530 is configured to adjust the residual value in the first residual image to determine a second residual image. The fourth determination unit 1540 is configured to determine a second reconstructed image based on the first reconstructed image and the second residual image.

[0212] In some implementations, the decoder 1500 further includes: a first writing unit configured to write first identification information into a bitstream, the first identification information including a first value and a second value, the first value being used to indicate that residual adjustment is to be performed on the first residual image, and the second value being used to indicate that residual adjustment is not to be performed on the first residual image.

[0213] In some implementations, if the rate-distortion cost corresponding to the second reconstructed image is less than the rate-distortion cost corresponding to the first reconstructed image, the value of the first identification information is the first value; if the rate-distortion cost corresponding to the second reconstructed image is greater than or equal to the rate-distortion cost corresponding to the first reconstructed image, the value of the first identification information is the second value.

[0214] In some implementations, the decoder 1500 further includes: a second writing unit configured to write second identification information into the bitstream, where the second identification information is used to indicate whether to perform residual adjustment on the image sequence.

[0215] In some implementations, the second identification information includes a third value and a fourth value, the third value is used to indicate that residual adjustment is performed on the image sequence, and the fourth value is used to indicate that residual adjustment is not performed on the image sequence.

[0216] In some implementations, the decoder 1500 further includes: a third writing unit configured to write first index information into the bitstream, where the first index information is used to indicate that the first residual image uses a target residual adjustment method among multiple residual adjustment methods.

[0217] In some implementations, the second residual image includes multiple residual images determined based on the multiple residual adjustment methods, and the target residual adjustment method is determined based on rate-distortion costs corresponding to the multiple residual images.

[0218] In some implementations, the first index information is used to indicate a first offset value, and the residual value of the second residual image is determined based on the first offset value and the residual value of the first residual image.

[0219] In some implementations, the first index information belongs to supplemental enhancement information.

[0220] In some implementations, the first residual image includes a first residual value, the second residual image includes a second residual value corresponding to the first residual value, and the first residual value and the second residual value satisfy at least one of the following:

[0221] If the first residual value is a positive number, the second residual value is equal to the difference between the first residual value and the first offset value;

[0222] If the first residual value is a negative number, the second residual value is equal to the sum of the first residual value and the first offset value;

[0223] If the first residual value is 0, the second residual value is equal to the first residual value.

[0224] In some implementations, the first bias value includes a plurality of bias values ​​having different values, and the plurality of bias values ​​are respectively used to adjust residual values ​​within a plurality of value ranges.

[0225] In some implementations, the neural network is a neural network-based super-resolution analyzer.

[0226] In some implementations, the first determining unit 1510 is configured to upsample the third reconstructed image to determine the first reconstructed image.

[0227] In some implementations, the second determining unit 1520 is configured to input the third reconstructed image into the neural network to determine the first residual image, wherein a resolution of the first residual image is higher than a resolution of the third reconstructed image.

[0228] In some implementations, the neural network is a neural network-based post-filter.

[0229] In some implementations, the second determining unit 1520 is configured to: input the first reconstructed image into the neural network to determine the first residual image.

[0230] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0231] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0232] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 1500. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the decoding method described in any one of the aforementioned embodiments.

[0233] Based on the composition of the above-mentioned encoder and the computer-readable storage medium, refer to Figure 16, which shows a schematic diagram of the specific hardware structure of the encoder provided by an embodiment of the present application. As shown in Figure 16, the encoder 1600 may include: a communication interface 1610, a memory 1620 and a processor 1630; each component is coupled together through a bus system 1640. It can be understood that the bus system 1640 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1640 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 1640 in Figure 16. Among them,

[0234] Communication interface 1610, used for sending and receiving signals during the process of sending and receiving information with other external network elements;

[0235] Memory 1620, for storing computer programs;

[0236] The processor 1630 is configured to, when running the computer program, execute:

[0237] determining a first reconstructed image;

[0238] determining a first residual image corresponding to the first reconstructed image according to a neural network;

[0239] adjusting the residual value in the first residual image to determine a second residual image;

[0240] A second reconstructed image is determined according to the first reconstructed image and the second residual image.

[0241] It is understood that the memory 1620 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1620 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0242] The processor 1630 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1630. The above-mentioned processor 1630 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1620 , and the processor 1630 reads the information in the memory 1620 and completes the steps of the above method in combination with its hardware.

[0243] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0244] Optionally, as another embodiment, the processor 1630 is further configured to execute the encoding method in the aforementioned embodiment when running the computer program.

[0245] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0246] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0247] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0248] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0249] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0250] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A decoding method, applied to a decoder, includes: Analyze a bitstream to determine a first reconstructed image; Determine a first residual image corresponding to the first reconstructed image according to a neural network; Adjust the residual values in the first residual image to determine a second residual image; Determine a second reconstructed image according to the first reconstructed image and the second residual image.

2. The method according to claim 1, wherein The method further includes: Analyze the bitstream to determine first identification information, where the first identification information is used to indicate whether to perform residual adjustment on the first residual image.

3. The method according to claim 2, wherein, The first identification information includes a first value and a second value, where the first value is used to indicate performing residual adjustment on the first residual image, and the second value is used to indicate not performing residual adjustment on the first residual image.

4. The method according to claim 1, wherein The method further includes: Analyze the bitstream to determine second identification information, where the second identification information is used to indicate whether to perform residual adjustment on an image sequence.

5. The method according to claim 4, wherein, The second identification information includes a third value and a fourth value, where the third value is used to indicate performing residual adjustment on the image sequence, and the fourth value is used to indicate not performing residual adjustment on the image sequence.

6. The method according to claim 1, wherein The method further includes: Analyze the bitstream to determine first index information, where the first index information is used to indicate a residual adjustment method for the first residual image.

7. The method according to claim 6, wherein, The first index information is used to indicate a first offset value, and the residual values of the second residual image are determined based on the first offset value and the residual values of the first residual image.

8. The method according to claim 6, wherein The first index information belongs to supplementary enhancement information.

9. The method according to any one of claims 1 to 8, wherein, The first residual image includes first residual values, the second residual image includes second residual values corresponding to the first residual values, and the first residual values and the second residual values satisfy at least one of the following: If the first residual value is positive, the second residual value is equal to the difference between the first residual value and the first offset value; If the first residual value is negative, the second residual value is equal to the sum of the first residual value and the first offset value; If the first residual value is 0, the second residual value is equal to the first residual value.

10. The method according to claim 8 or 9, wherein The first offset value includes multiple offset values with different values, and the multiple offset values are respectively used to adjust residual values within multiple value ranges.

11. The method according to any one of claims 1 to 10, wherein, The neural network is a neural network-based superresolver.

12. The method according to any one of claims 1 to 11, wherein, The step of analyzing the bitstream to determine a first reconstructed image includes: Analyze the bitstream to determine a third reconstructed image; Upsample the third reconstructed image to determine the first reconstructed image.

13. The method according to claim 12, wherein, The step of determining a first residual image corresponding to the first reconstructed image according to a neural network includes: Input the third reconstructed image into the neural network to determine the first residual image, where the resolution of the first residual image is higher than the resolution of the third reconstructed image.

14. The method according to any one of claims 1 to 10, wherein The neural network is a neural network-based postfilter.

15. The method according to any one of claims 1 to 10 and 14, wherein The step of determining a first residual image corresponding to the first reconstructed image according to a neural network includes: Input the first reconstructed image into the neural network to determine the first residual image.

16. An encoding method, applied to an encoder, includes: Determine a first reconstructed image; Determine a first residual image corresponding to the first reconstructed image according to a neural network; Adjust the residual values in the first residual image to determine a second residual image; Determine a second reconstructed image based on the first reconstructed image and the second residual image.

17. The method according to claim 16, wherein, The method further includes: Write first identification information into a bitstream, where the first identification information includes a first value and a second value, the first value is used to indicate that residual adjustment is performed on the first residual image, and the second value is used to indicate that no residual adjustment is performed on the first residual image.

18. The method according to claim 17, wherein: If the rate-distortion cost corresponding to the second reconstructed image is less than the rate-distortion cost corresponding to the first reconstructed image, the value of the first identification information is the first value; and / or If the rate-distortion cost corresponding to the second reconstructed image is greater than or equal to the rate-distortion cost corresponding to the first reconstructed image, the value of the first identification information is the second value.

19. The method according to claim 16, wherein, The method further includes: Write second identification information into the bitstream, where the second identification information is used to indicate whether residual adjustment is performed on an image sequence.

20. The method according to claim 19, wherein The second identification information includes a third value and a fourth value, the third value is used to indicate that residual adjustment is performed on the image sequence, and the fourth value is used to indicate that no residual adjustment is performed on the image sequence.

21. The method according to claim 16, wherein The method further includes: Write first index information into the bitstream, where the first index information is used to indicate a target residual adjustment method among multiple residual adjustment methods for the first residual image.

22. The method according to claim 21, wherein The second residual image includes multiple residual images determined based on the multiple residual adjustment methods, and the target residual adjustment method is determined based on the rate-distortion costs corresponding to the multiple residual images.

23. The method according to claim 21, wherein, The first index information is used to indicate a first offset value, and the residual value of the second residual image is determined based on the first offset value and the residual value of the first residual image.

24. The method according to any one of claims 21 to 23, wherein The first index information belongs to supplementary enhancement information.

25. The method according to any one of claims 16 to 24, wherein The first residual image includes a first residual value, the second residual image includes a second residual value corresponding to the first residual value, and the first residual value and the second residual value satisfy at least one of the following: If the first residual value is positive, the second residual value is equal to the difference between the first residual value and the first offset value; If the first residual value is negative, the second residual value is equal to the sum of the first residual value and the first offset value; If the first residual value is 0, the second residual value is equal to the first residual value.

26. The method according to claim 23 or 25, wherein The first offset value includes multiple offset values with different values, and the multiple offset values are respectively used to adjust the residual values within multiple value ranges.

27. The method according to any one of claims 16 to 26, wherein The neural network is a neural network-based superresolver.

28. The method according to any one of claims 16 to 27, wherein The determining of the first reconstructed image includes: Upsample a third reconstructed image to determine the first reconstructed image.

29. The method according to claim 28, wherein, The determining of the first residual image corresponding to the first reconstructed image according to the neural network includes: Input the third reconstructed image into the neural network to determine the first residual image, where the resolution of the first residual image is higher than the resolution of the third reconstructed image.

30. The method according to any one of claims 16 to 26, wherein The neural network is a neural network-based postfilter.

31. The method according to any one of claims 16 to 26 and 30, wherein, Determining the first residual image corresponding to the first reconstructed image according to the neural network includes: Inputting the first reconstructed image into the neural network to determine the first residual image.

32. A decoder, the decoder includes: A first determination unit configured to parse a bitstream and determine a first reconstructed image; A second determination unit configured to determine a first residual image corresponding to the first reconstructed image according to a neural network; A third determination unit configured to adjust the residual values in the first residual image to determine a second residual image; A fourth determination unit configured to determine a second reconstructed image according to the first reconstructed image and the second residual image.

33. A decoder, the decoder includes: A memory for storing a computer program; A processor for executing the method according to any one of claims 1 to 15 when running the computer program.

34. An encoder, the encoder includes: A first determination unit configured to determine a first reconstructed image; A second determination unit configured to determine a first residual image corresponding to the first reconstructed image according to a neural network; A third determination unit configured to adjust the residual values in the first residual image to determine a second residual image; A fourth determination unit configured to determine a second reconstructed image according to the first reconstructed image and the second residual image.

35. An encoder, the encoder includes: A memory for storing a computer program; A processor for executing the method according to any one of claims 16 to 31 when running the computer program.

36. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the method according to any one of claims 1 to 15, or the method according to any one of claims 16 to 31.

37. A non-volatile computer-readable storage medium storing a bitstream, the bitstream being generated by an encoding method using an encoder or being decoded by a decoding method using a decoder, wherein, The decoding method is the method according to any one of claims 1 to 15, and the encoding method is the method according to any one of claims 16 to 31.

Images