Encoding method, decoding method, encoder, decoder, code stream, and storage medium

Abstract

The present application provides an encoding method, a decoding method, an encoder, a decoder, a code stream, and a storage medium. The decoding method comprises: parsing a code stream to determine a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image on the basis of the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; and determining a third reconstructed image on the basis of the first reconstructed image and the second residual image.

Description

Coding and decoding method, codec, code stream and storage medium Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to a coding and decoding method, a codec, a bit stream, and a storage medium. Background Art

[0002] In video codec frameworks, the output information of loop filter modules can be inaccurate. For example, in an adaptive loop filter (ALF), the filter coefficient set consists of a fixed subset and an adaptive parameter set (APS) subset. The fixed subset is obtained through offline training on training set images, while the APS subset is obtained through online training on the current encoded image. The trained filter coefficient set may suffer from overfitting or underfitting issues.

[0003] Therefore, how to improve the accuracy of the output information of the loop filtering module to improve the encoding and decoding performance is a problem that needs to be solved.

[0004] Summary of the Invention

[0005] The present application provides a coding and decoding method, a codec, a bit stream, and a storage medium. The following introduces various aspects of the present application.

[0006] In a first aspect, a decoding method is provided, which is applied to a decoder, and the decoding method includes: parsing a code stream to determine a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image based on the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; and determining a third reconstructed image based on the first reconstructed image and the second residual image.

[0007] In a second aspect, a coding method is provided, which is applied to an encoder and includes: determining a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image based on the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; and determining a third reconstructed image based on the first reconstructed image and the second residual image.

[0008] According to a third aspect, a decoder is provided, comprising: a first determination unit configured to parse a code stream and determine a first reconstructed image; a second determination unit configured to perform a loop filtering operation on the first reconstructed image and determine a second reconstructed image; a third determination unit configured to determine a first residual image based on the first reconstructed image and the second reconstructed image; a fourth determination unit configured to perform residual adjustment on the first residual image and determine a second residual image; and a fifth determination unit configured to determine a third reconstructed image based on the first reconstructed image and the second residual image.

[0009] In a fourth aspect, a decoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method of the first aspect when running the computer program.

[0010] In a fifth aspect, an encoder is provided, comprising: a first determination unit configured to determine a first reconstructed image; a second determination unit configured to perform a loop filtering operation on the first reconstructed image to determine a second reconstructed image; a third determination unit configured to determine a first residual image based on the first reconstructed image and the second reconstructed image; a fourth determination unit configured to perform residual adjustment on the first residual image to determine the second residual image; and a fifth determination unit configured to determine a third reconstructed image based on the first reconstructed image and the second residual image.

[0011] In a sixth aspect, an encoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method of the second aspect when running the computer program.

[0012] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the method of the first aspect or the second aspect is implemented.

[0013] In an eighth aspect, a computer program product is provided, comprising a computer program, which implements the method of the first aspect or the second aspect when the computer program is executed.

[0014] In a ninth aspect, a non-volatile computer-readable storage medium for storing a bit stream is provided, wherein the bit stream is generated by an encoding method of an encoder, or the bit stream is decoded by a decoding method of a decoder, wherein the decoding method is the method described in the first aspect and the encoding method is the method described in the second aspect.

[0015] In a tenth aspect, a code stream is provided, comprising a code stream generated according to the method of the second aspect.

[0016] The embodiment of the present application performs residual adjustment on the image obtained after the loop filtering operation, thereby potentially improving the accuracy of the output information of the loop filtering module and further improving the encoding and decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a structural diagram illustrating an example of a video encoder to which an embodiment of the present application may be applied.

[0018] FIG2 is a diagram showing an example structure of a video decoder to which an embodiment of the present application can be applied.

[0019] FIG3 is an exemplary diagram of a loop filtering module to which an embodiment of the present application can be applied.

[0020] FIG4 is a flow chart of a decoding method provided in an embodiment of the present application.

[0021] FIG5A is an example diagram of a residual image.

[0022] FIG5B is another example diagram of a residual image.

[0023] FIG6 is a flow chart of the encoding method provided in an embodiment of the present application.

[0024] FIG7 is a schematic diagram of the structure of the residual information adjustment module provided in an embodiment of the present application.

[0025] FIG8 is a schematic diagram of the input and output of the residual information adjustment module provided in an embodiment of the present application.

[0026] FIG9 is a schematic diagram of residual adjustment based on coding characteristics provided by an implementation of the present application.

[0027] FIG10 is a schematic diagram of the structure of a decoder provided in one embodiment of the present application.

[0028] FIG11 is a schematic diagram of the structure of a decoder provided in another embodiment of the present application.

[0029] FIG12 is a schematic diagram of the structure of an encoder provided in one embodiment of the present application.

[0030] FIG13 is a schematic diagram of the structure of an encoder provided in another embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solution in this application will be described below with reference to the accompanying drawings.

[0032] FIG1 is a schematic block diagram of a video encoder according to an embodiment of the present application.

[0033] It should be understood that the video encoder 100 can be used to perform lossy compression or lossless compression on an image. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0034] The video encoder 100 can be applied to image data in a luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0035] For example, the video encoder 100 reads video data and, for each image in the video data, divides the image into a number of coding tree units (CTUs). In some examples, a CTU may be referred to as a "tree block," "largest coding unit" (LCU) or "coding tree block" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to one luminance (luminance or luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding. A CU may be a rectangular block or a square block. A CU may correspond to a prediction unit (PU) and a transform unit (TU).

[0036] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0037] In some embodiments, as shown in FIG1 , the video encoder 100 may include a prediction module 110, a residual module 120, a transform / quantization module 130, an inverse transform / quantization module 140, a reconstruction module 150, a loop filter module 160, a decoded image buffer 170, and an entropy coding module 180. It should be noted that the video encoder 100 may include more, fewer, or different functional components.

[0038] Optionally, in this application, the current block may be referred to as the current coding unit (CU). The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstruction block. Due to the need for parallel processing, an image may be divided into slices. Slices in the same image may be processed in parallel, meaning that there is no data dependency between them. The term "frame" is commonly used, and it can generally be understood that a frame is an image. The term "frame" herein may also be replaced by "image" or "slice," etc.

[0039] In some embodiments, the prediction module 110 includes an inter-frame prediction module 111 and an intra-frame prediction module 112. Because there is a strong correlation between adjacent pixels in a video image, intra-frame prediction is used in video coding and decoding technologies to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent images in a video, inter-frame prediction is used in video coding and decoding technologies to eliminate temporal redundancy between adjacent images, thereby improving coding efficiency.

[0040] The inter-frame prediction module 111 can be used for inter-frame prediction. Inter-frame prediction can include motion estimation and motion compensation. It can refer to image information from different images. Inter-frame prediction uses motion information to find a reference block from a reference image and generate a prediction block based on the reference block to eliminate temporal redundancy. Inter-frame prediction uses motion information to find a reference block from a reference image and generate a prediction block based on the reference block. Motion information includes the reference image list in which the reference image is located, the reference image index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference image. Here, the integer pixel or fractional pixel block in the reference image found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Reprocessing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.

[0041] The intra-frame prediction module 112 only refers to information of the same image to predict pixel information within the current code image block to eliminate spatial redundancy.

[0042] Intra-frame prediction has multiple prediction modes. For example, the H-series international digital video coding standard H.264 / AVC has eight angular prediction modes and one non-angular prediction mode. H.265 / HEVC expands this to 33 angular prediction modes and two non-angular prediction modes. High-efficiency video coding (HEVC) uses planar, direct current (DC), and 33 angular modes for a total of 35 intra-frame prediction modes. Versatile video coding (VVC) uses planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes.

[0043] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of high-definition and ultra-high-definition digital video development.

[0044] Residual module 120 may generate a residual block for a CU based on the pixel block of the CU and the prediction block of the CU. For example, residual module 120 may generate a residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel block of the CU and the corresponding sample in the prediction block of the CU.

[0045] The transform / quantization module 130 may quantize the transform coefficients. The transform / quantization module 130 may quantize the transform coefficients associated with the CU based on a quantization parameter (QP) value associated with the CU. The video encoder 100 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0046] The inverse transform / quantization module 140 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0047] Reconstruction module 150 can add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction module 110 to generate a reconstructed image block associated with the CU. By reconstructing each sample block of the CU in this manner, video encoder 100 can reconstruct the pixel blocks of the CU.

[0048] The loop filter module 160 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent pixel encoding. For example, it can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with the CU.

[0049] In some embodiments, the loop filtering module 160 includes a deblocking filtering module and a sample adaptive offset / adaptive loop filtering (SAO / ALF) module, wherein the deblocking filtering module is used to remove blocking effects, and the SAO / ALF module is used to remove ringing effects.

[0050] The decoded image buffer 170 may store the reconstructed pixel blocks. The inter prediction module 111 may use a reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra prediction module 112 may use the reconstructed pixel blocks in the decoded image buffer 170 to perform intra prediction on other PUs in the same image as the CU.

[0051] The entropy encoding module 180 may receive the quantized transform coefficients from the transform / quantization module 130. The entropy encoding module 180 may perform one or more entropy encoding operations on the quantized transform coefficients to generate entropy-encoded data.

[0052] FIG2 is a schematic block diagram of a video decoder according to an embodiment of the present application.

[0053] 2 , video decoder 200 includes an entropy decoding module 210, a prediction module 220, an inverse quantization / transformation module 230, a reconstruction module 240, a loop filter module 250, and a decoded image buffer 260. It should be noted that video decoder 200 may include more, fewer, or different functional components.

[0054] The video decoder 200 may receive a bitstream. The entropy decoding module 210 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding module 210 may parse the entropy-encoded syntax elements in the bitstream. The prediction module 220, the inverse quantization / transformation module 230, the reconstruction module 240, and the loop filter module 250 may decode the video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.

[0055] In some embodiments, the prediction module 220 includes an intra-frame prediction module 222 and an inter-frame prediction module 221 .

[0056] The intra prediction module 222 may perform intra prediction to generate a prediction block for a PU. The intra prediction module 222 may use an intra prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. The intra prediction module 222 may also determine the intra prediction mode for the PU based on one or more syntax elements parsed from the codestream.

[0057] The inter-frame prediction module 221 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. Furthermore, if a PU is encoded using inter-frame prediction, the entropy decoding module 210 may parse the motion information of the PU. The inter-frame prediction module 221 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction module 221 may generate a prediction block for the PU based on the one or more reference blocks of the PU.

[0058] The inverse quantization / transform module 230 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform module 230 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0059] After inverse quantizing the transform coefficients, inverse quantization / transform module 230 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0060] Reconstruction module 240 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, reconstruction module 240 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0061] The loop filtering module 250 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0062] The video decoder 200 may store the reconstructed image of the CU in the decoded image buffer 260. The video decoder 200 may use the reconstructed image in the decoded image buffer 260 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0063] The basic process of video encoding and decoding is as follows: At the encoder end, an image is divided into blocks. For the current block, the prediction module 110 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual module 120 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block can also be referred to as residual information. This residual block undergoes transformation and quantization by the transform / quantization module 130, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization module 130 can be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization module 130 can be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding module 180 receives the quantized change coefficients output by the transform and quantization module 130 and performs entropy coding on these quantized change coefficients to output a bitstream. For example, the entropy coding module 180 can eliminate character redundancy based on the target context model and the probability information of the binary bitstream.

[0064] At the decoding end, the entropy decoding module 210 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction module 220 uses intra-frame prediction or inter-frame prediction on the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation module 230 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction module 240 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering module 250 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference image for inter-frame prediction of subsequent images.

[0065] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and analyzes the existing information to determine the same block division information, prediction, transform, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical.

[0066] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0067] As described above, the loop filtering module performs loop filtering on the reconstructed image based on an image or a block to obtain a filtered reconstructed image. As shown in Figure 3, the loop filtering module may include filtering units such as luma mapping with chroma scaling (LMCS) 310, deblocking filter (DBF) 320, sample adaptive offset (SAO) 330, adaptive loop filter (ALF) 340, and cross component adaptive loop filter (CCALF) 350. The following is a detailed introduction to the ALF and CCALF in the loop filtering module.

[0068] ALF includes luma ALF and chroma ALF. Luma ALF is designed to use luma information to minimize the mean square error (MSE) between the luma reconstructed image and the original image, while chroma ALF is designed to use chroma information to minimize the mean square error (MSE) between the chroma reconstructed image and the original image. CCALF, on the other hand, is a filter designed to use luma information to minimize the mean square error (MSE) between the chroma reconstructed image and the original image. ALF / CCALF is based on the Wiener filtering principle and establishes the Wiener-Hoff equation using the original image information and the reconstructed image information. This equation is then solved to obtain a series of filter coefficients with the minimum mean square error. Applying these filter coefficients to the filtering process of the reconstructed image helps reduce decoding errors, thereby improving coding performance.

[0069] ALF's filter coefficient set consists of a fixed subset and an APS subset, while CCALF specifies that only the APS subset can be used. The fixed subset is a pre-trained filter coefficient subset, specified in the standard and not required to be transmitted. The APS subset is generated based on the current reconstructed image using the Wiener filtering principle. The APS subset must be transmitted from the encoder to the decoder via the APS. For the ALF's APS subset, the ALF classifies each coding tree unit (CTU) (all pixels in a CTU belong to the same class), accumulates the covariance matrix and error vector for pixels in the same class, and then constructs the Wiener-Hopper equation to calculate the ALF filter coefficients by solving it. For the CCALF's APS subset, the CCALF divides the image into an integer number of regions, each of which contains a consecutive integer number of CTUs. CCALF then calculates the filter coefficients for each region.

[0070] Taking ALF as an example, during encoding, the ALF determines whether to use a fixed subset or an APS subset based on rate-distortion optimization. It then selects a filter of the corresponding category based on the filter coefficients for filtering, and then writes the filtered result into the reconstructed image. Furthermore, if the ALF selects the APS subset, the filter coefficients must also be written into the APS, and the corresponding syntax elements must be written into the bitstream through entropy coding.

[0071] In the video codec framework, the output information of the current adaptive loop filter module may be inaccurate. The fixed subset is obtained through offline training based on the training set images, while the APS subset is obtained through online training based on the current coded image. A set of filter coefficient subsets obtained through offline or online training will be used for the same type of image blocks or regions. In training-based methods, there is no guarantee that the training results will be optimal. Training may result in a local optimal solution and may also suffer from overfitting or underfitting problems. In other words, using the trained filter coefficients to filter the reconstructed image does not guarantee the accuracy of the reconstructed image.

[0072] In response to the above problems, an embodiment of the present application provides an encoding method, including: determining a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image based on the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; and determining a third reconstructed image based on the first reconstructed image and the second residual image.

[0073] In addition, an embodiment of the present application also provides a decoding method, including: parsing a code stream to determine a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image based on the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; and determining a third reconstructed image based on the first reconstructed image and the second residual image.

[0074] The embodiment of the present application performs residual adjustment on the image obtained after the loop filtering operation, thereby potentially improving the accuracy of the output information of the loop filtering module and further improving the encoding and decoding performance.

[0075] The decoding method of the embodiment of the present application is described in detail below with reference to FIG4 .

[0076] Figure 4 is a flowchart of a decoding method provided by an embodiment of the present application. The method of Figure 4 can be applied to a decoder.

[0077] Referring to FIG. 4 , in step S410 , the code stream is parsed to determine a first reconstructed image.

[0078] In some implementations, determining the first reconstructed image may include: performing inverse quantization and inverse transformation on quantization coefficients to determine a residual image; and using the sum of the residual image and the predicted image as the first reconstructed image.

[0079] In step S420 , a loop filtering operation is performed on the first reconstructed image to determine a second reconstructed image.

[0080] The embodiments of the present application do not specifically limit the loop filtering operation. In some implementations, the loop filtering operation may include at least one of the following: luma mapping and chroma scaling (LMCS), deblocking filtering (DBF), sample offset compensation (SAO), adaptive loop filtering (ALF), and cross-component adaptive loop filtering (CCALF). For example, the loop filtering operation may be an operation performed in adaptive loop filtering.

[0081] In step S430 , a first residual image is determined according to the first reconstructed image and the second reconstructed image.

[0082] It should be noted that the image (such as a reconstructed image or a residual image) mentioned in the embodiment of the present application may refer to a frame of image, or a slice in a frame of image, or a CTU, or a CU (or coding block). In addition, the image mentioned in the embodiment of the present application may refer to an image corresponding to a component (color or brightness), or an image corresponding to multiple components. Taking an image including three components of Y, Cb, and Cr as an example, the image mentioned in the embodiment of the present application may refer to an image corresponding to the Y component, an image corresponding to the Cb component, or an image corresponding to the Cr component.

[0083] The first residual image mentioned in step S430 may include residual information of the first reconstructed image and the second reconstructed image. For example, the first residual image may be obtained by taking the difference between the pixel values ​​of corresponding pixel positions of the first reconstructed image and the second reconstructed image.

[0084] In step S440 , residual adjustment (RA) is performed on the first residual image to determine a second residual image.

[0085] When performing residual adjustment on the first residual image, the residual adjustment may be performed on all residuals in the first residual image, or on part of the residuals in the first residual image, which is not specifically limited in the embodiments of the present application. The residual adjustment here may refer to increasing the residual value or decreasing the residual value.

[0086] In the embodiment of the present application, a certain residual adjustment formula can be used to adjust the residual value in the first residual image to obtain the second residual image. Alternatively, one or more fixed offset values ​​(which can be positive numbers) can be used to adjust the residual value in the first residual image to obtain the second residual image.

[0087] In the embodiment of the present application, the first reconstructed image can be adjusted based on multiple residual adjustment methods. Different residual adjustment methods in the multiple residual adjustment methods process positive residual values ​​and / or negative residual values ​​in the first residual image in different ways. The multiple residual adjustment methods are described in detail below.

[0088] Residual adjustment method 1

[0089] The following assumes that the first residual image includes a first residual value, and the second residual image includes a second residual value corresponding to the first residual value (that is, the second residual value is a residual value obtained after adjusting the first residual value). The following is introduced using an example of adjusting the residual value of the first residual image using a fixed first offset value.

[0090] For example, since the residual values ​​in a residual image can be either positive or negative, the first offset value can be adjusted so that the first and second residual values ​​satisfy at least one of the following conditions: if the first residual value is positive, the second residual value is equal to the difference between the first residual value and the first offset value; if the first residual value is negative, the second residual value is equal to the sum of the first residual value and the first offset value; if the first residual value is 0, no adjustment is performed. After this adjustment, the overall absolute residual value of the first residual image will decrease. Testing has found that in most cases, the residual values ​​in the first residual image are larger than the actual residual values. Therefore, reducing the residual values ​​of the first residual image helps improve the accuracy of the reconstructed image. Taking Figures 5A and 5B as examples, Figure 5A shows an example of the first residual image, and Figure 5B shows an example of the second residual image. The first offset value can be set to 1 to adjust the residual value at each pixel position in Figure 5A. For residual values ​​greater than 0, an offset value of 1 is subtracted; for residual values ​​less than 0, an offset value of 1 is added; for residual values ​​equal to 0, no adjustment is performed. After adjustment, the residual value shown in Figure 5B is obtained. Using a fixed offset value to adjust the residual value is very simple to implement and does not increase decoding complexity.

[0091] As an example, a pseudo code for performing residual adjustment on the first residual image may be as follows:

[0092] if(RecAfterALF–RecBeforeALF>0)

[0093] Rec_new = RecAfterALF - 1;

[0094] else if (RecAfterALF – RecBeforeALF < 0)

[0095] Rec_new = RecAfterALF + 1;

[0096] else

[0097] Rec_new = RecAfterALF;

[0098] For another example, the adjustment of the first offset value can make the first residual value and the second residual value satisfy at least one of the following: if the first residual value is positive, the second residual value is equal to the sum of the first residual value and the first offset value; if the first residual value is negative, the second residual value is equal to the difference between the first residual value and the first offset value.

[0099] For another example, the adjustment of the first offset value can make the first residual value and the second residual value satisfy: if the first residual value is not 0, the second residual value is equal to the sum of the first residual value and the first offset value.

[0100] For another example, the adjustment of the first offset value can make the first residual value and the second residual value satisfy: if the first residual value is not 0, the second residual value is equal to the difference between the first residual value and the first offset value.

[0101] Residual adjustment method two

[0102] Adjust the residual values within a specific value range in the first residual image.

[0103] For example, the first residual image includes a third residual value. If the third residual value belongs to at least one value range, perform residual adjustment on the third residual value; if the third residual value does not belong to at least one value range, do not perform residual adjustment on the third residual value.

[0104] As an example, perform local residual adjustment within the value range (minimum value min to maximum value max), where both min and max are positive numbers. For a positive residual value res, perform adjustment only when (min < res < max); for a negative residual value res, perform adjustment only when (-max < res < -min).

[0105] Residual adjustment method three

[0106] The residuals in different value ranges in the first residual image are adjusted using different offset values.

[0107] Multiple offset values ​​can be understood as multiple offset values ​​of different precisions. For larger residual values, a lower precision offset value (a larger value) can be used to adjust them; for smaller residual values, a higher precision offset value (a smaller value) can be used to adjust them.

[0108] As an example, assuming that for the current pixel of the current slice, the corresponding residual value is res, and the offset value corresponding to res is RA_FACTOR, the value of RA_FACTOR can be derived using the following strategy (expressed in pseudo code):

[0109] Among them, {x1, x2, x3} represents a positive residual value, {y1, y2, y3} represents a negative residual value, and {a1, a2, a3} and {b1, b2, b3} are both preset candidate offset values.

[0110] It should be understood that the above-mentioned various residual adjustment methods can be used individually or in combination. The offset value in the various residual adjustment methods can be a single offset value or a set of offset values ​​formed by multiple offset values ​​with different values. The multiple offset values ​​can be used to adjust the residual values ​​within multiple value ranges.

[0111] In step S450 , a third reconstructed image is determined according to the first reconstructed image and the second residual image.

[0112] In some implementations, step S450 may include: taking the sum of pixel values ​​at corresponding pixel positions of the first reconstructed image and the second residual image as the pixel value of the third reconstructed image.

[0113] The embodiment of the present application performs residual adjustment on the image obtained after the loop filtering operation, thereby potentially improving the accuracy of the output information of the loop filtering module and further improving the decoding performance.

[0114] The above describes in detail how to adjust the residual image determined based on loop filtering. The adjustment of the residual image by the decoder can be based on auxiliary information (such as syntax elements) in the code stream. The following is a detailed example of the auxiliary information that may be used to implement the residual image adjustment scheme at the decoder. It should be understood that the auxiliary information mentioned below is optional information. In some implementations, the codec can adjust the residual image according to the same predefined rules without the need for such auxiliary information.

[0115] In some implementations, the code stream may carry first identification information. Therefore, during the decoding process, the code stream may be parsed to determine the first identification information. The first identification information may be used to indicate whether residual adjustment is performed (or whether residual adjustment is allowed). The first identification information may include a first value and a second value. The first value may be, for example, 1 or true. The second value may be, for example, 0 or false. If the value of the first identification information is the first value, the decoding end is instructed to perform residual adjustment; if the value of the first identification information is 0, the decoding end may be instructed not to perform residual adjustment.

[0116] The first identification information may refer to identification information at a slice level; or, the first identification information may refer to identification information at a coding tree unit (CTU) level; or, the first identification information may refer to identification information at a coding unit (CU) level.

[0117] For example, if the reconstructed image belongs to a slice-level luminance component, the first identification information may be set to sh_ra_enabled_flag. The value of sh_ra_enabled_flag may include 0 and 1. If the value of sh_ra_enabled_flag is 1, it indicates that residual adjustment is performed on the reconstructed image; if the value of sh_ra_enabled_flag is 0, it indicates that residual adjustment is not performed on the reconstructed image.

[0118] For another example, if the reconstructed image belongs to the slice-level chroma Cb component, the first identification information can be set to sh_ra_cb_enabled_flag. The value of sh_ra_cb_enabled_flag can include 0 and 1. If the value of sh_ra_cb_enabled_flag is 1, it indicates that residual adjustment is performed on the reconstructed image; if the value of sh_ra_cb_enabled_flag is 0, it indicates that residual adjustment is not performed on the reconstructed image.

[0119] For another example, if the reconstructed image belongs to the slice-level chroma Cr component, the first identification information can be set to sh_ra_cr_enabled_flag. The value of sh_ra_cr_enabled_flag can include 1 and 0. If the value of sh_ra_cr_enabled_flag is 1, it indicates that residual adjustment is performed on the reconstructed image; if the value of sh_ra_cr_enabled_flag is 0, it indicates that residual adjustment is not performed on the reconstructed image.

[0120] Assuming that the first identification information is identification information at a slice level, the second identification information may be set in a syntax element at a coding tree unit (CTU) level or a coding unit (CU) level.

[0121] In some implementations, the code stream may carry second identification information. Therefore, during the decoding process, the code stream may be parsed to determine the second identification information. The second identification information may be used to indicate whether residual adjustment is performed on the coding tree unit (or coding unit) (or whether residual adjustment is allowed on the coding tree unit). The second identification information may include a third value and a fourth value. The third value may be, for example, 1 or true. The fourth value may be, for example, 0 or false. If the value of the second identification information is the third value, it indicates that the decoding end performs residual adjustment on the coding tree unit; if the value of the second identification information is the fourth value, it may indicate that the decoding end does not perform residual adjustment on the coding tree unit.

[0122] For example, if the reconstructed image belongs to the block-level luminance component, the second identification information can be set to ra_ctb_enabled_flag[x][y]. The value of ra_ctb_enabled_flag[x][y] can include 1 and 0. If the value of ra_ctb_enabled_flag[x][y] is 1, it indicates that residual adjustment is performed on the coding tree unit; if the value of ra_ctb_enabled_flag[x][y] is 0, it indicates that residual adjustment is not performed on the coding tree unit.

[0123] It should be understood that the second identification information and the first identification information mentioned above can be used in combination. For example, when the value of the first identification information is the first value (i.e., residual adjustment of the current slice is allowed), the value of the second identification information can continue to be judged. If the value of the second identification information is the third value (i.e., residual adjustment of the coding tree unit is allowed), the current coding tree unit is adjusted; if the value of the second identification information is the fourth value (i.e., residual adjustment of the coding tree unit is not allowed), the current coding tree unit is not adjusted. For another example, when the value of the first identification information is the second value (i.e., residual adjustment of the current slice is not allowed), the residual adjustment of the image in the current coding tree unit is not performed. In this case, the decoder does not need to parse the second identification information, and the default value of the second identification information of each coding tree unit in the current slice is the fourth value.

[0124] As mentioned above, the residual value can be adjusted based on multiple offset values. Therefore, in some implementations, the code stream is parsed to determine the first index information. The first index information is used to indicate the first offset value from multiple candidate offset values. For example, two candidate offset values ​​can be pre-set: 1 and 2. Among them, the value of the first index information corresponding to the candidate offset value 1 is 0, and the value of the first index information corresponding to the candidate offset value 2 is 1. If the code stream is parsed and the value of the first index information is determined to be 1, it can be determined that the residual value of the first residual image is adjusted using the offset value 2. For example, if the first residual value in the first residual image is a positive number, the first residual value is subtracted by 2; if the first residual value in the first residual image is a negative number, the first residual value is added by 2; if the first residual value in the first residual image is 0, the first residual value remains unchanged.

[0125] The residual adjustment method used by the codec may include only one residual adjustment method or multiple residual adjustment methods. If only one residual adjustment method is included, when the decoder determines to adjust the first residual image, the residual adjustment method may be directly used for adjustment. If multiple residual adjustment methods are included, the code stream may carry index information indicating the residual adjustment method currently used. Still taking the residual adjustment of the first residual image as an example, the code stream may be parsed to determine the second index information. The second index information may be used to indicate the residual adjustment method of the first residual image. The second index information may indicate that the residual adjustment method of the first residual image is the first residual adjustment method among the preset multiple residual adjustment methods. For example, the preset residual adjustment methods include the above-mentioned residual adjustment method one and residual adjustment method two. The index of residual adjustment method one is 0, and the index of residual adjustment method two is 1. If the code stream is parsed and it is determined that the value of the second index information is 1, it can be determined that residual adjustment method two is used to adjust the residual value of the first residual image.

[0126] In some implementations, the first index information and / or the second index information may be used in combination with the first identification information mentioned above. For example, if the first identification information indicates that residual adjustment is to be performed, the first index information and / or the second index information may be parsed to determine the residual adjustment method. For another example, if the first identification information indicates that residual adjustment is not to be performed, the first index information and / or the second index information may not be parsed.

[0127] As mentioned above, the residual adjustment method can be determined based on the index information in the code stream. In addition, in some implementations, the residual adjustment method can be adaptively determined based on the coding parameters corresponding to the current block (which can refer to a coding tree unit (CTU) or a coding unit (CU)). The coding parameters here may include: the frame type corresponding to the current block (for example, I frame / P frame / B frame), the quantization parameter corresponding to the current block (for example, the QP of the general test condition or the QP of the slice level), and the filter coefficient set for loop filtering corresponding to the current block (for example, a fixed subset or an APS subset).

[0128] The following describes the test performance results obtained by testing the solution in the embodiments of the present application under the general test conditions Random Access.

[0129] Table 1 shows the performance test results of the embodiment of this application after residual adjustment of the ALF and CCALF combination in the loop filter module. It should be noted that for the luma component, the residual is calculated as the difference between the luma ALF output and the luma ALF input; for the chroma component, the residual is calculated as the difference between the chroma CCALF output and the chroma ALF input. In this test, the offset value is set to 1.

[0130] As can be seen from Table 1, in Class C and Class D, the test values ​​of the three components Y, U, and V are all reduced, which means that the solution provided in the embodiment of the present application can improve the encoding and decoding performance.

[0131] Table 1: Test performance under common test conditions: Random Access

[0132] Figure 6 is a flow chart of an encoding method provided by an embodiment of the present application. The method of Figure 6 can be applied to an encoder.

[0133] 6 , in step S610 , a first reconstructed image is determined.

[0134] In some implementations, determining the first reconstructed image may include: performing inverse quantization and inverse transformation on quantization coefficients to determine a residual image; and using the sum of the residual image and the predicted image as the first reconstructed image.

[0135] In step S620 , a loop filtering operation is performed on the first reconstructed image to determine a second reconstructed image.

[0136] The embodiments of the present application do not specifically limit the loop filtering operation. In some implementations, the loop filtering operation may refer to the operation of all submodules in the loop filtering module 160 in Figure 1. In other implementations, the loop filtering operation may refer to the operation of some submodules in the loop filtering module 160 in Figure 1. For example, the loop filtering operation may include at least one of the following: luminance mapping and chrominance scaling (LMCS), deblocking filtering (DBF), sample offset compensation (SAO), adaptive loop filtering (ALF), and cross-component adaptive loop filtering (CCALF). For example, the loop filtering operation may be an operation performed in adaptive loop filtering.

[0137] In step S630 , a first residual image is determined according to the first reconstructed image and the second reconstructed image.

[0138] It should be noted that the image (such as a reconstructed image or a residual image) mentioned in the embodiment of the present application may refer to a frame of image, or may refer to a slice (slice) in a frame of image, or a coding block. In addition, the image mentioned in the embodiment of the present application may refer to an image corresponding to a component (color or brightness), or may refer to an image corresponding to multiple components. Taking an image including three components Y, Cb, and Cr as an example, the image mentioned in the embodiment of the present application may refer to an image corresponding to the Y component, an image corresponding to the Cb component, or an image corresponding to the Cr component.

[0139] In some implementations, performing step S630 may include using the difference between the first reconstructed image and the second reconstructed image as the first residual image. For example, the difference between the pixel values ​​at corresponding pixel positions in the first reconstructed image and the second reconstructed image may be used as the pixel value of the first residual image.

[0140] In step S640 , residual adjustment is performed on the first residual image to determine a second residual image.

[0141] When performing residual adjustment on the first residual image, the residual adjustment may be performed on all residuals in the first residual image, or on part of the residuals in the first residual image, which is not specifically limited in the embodiments of the present application. The residual adjustment here may refer to increasing the residual value or decreasing the residual value.

[0142] In the embodiment of the present application, a certain residual adjustment formula can be used to adjust the residual value in the first residual image to obtain the second residual image. Alternatively, one or more fixed offset values ​​(which can be positive numbers) can be used to adjust the residual value in the first residual image to obtain the second residual image.

[0143] In the embodiment of the present application, the first reconstructed image can be adjusted based on multiple residual adjustment methods. Different residual adjustment methods in the multiple residual adjustment methods process positive residual values ​​and / or negative residual values ​​in the first residual image in different ways. The multiple residual adjustment methods are described in detail below.

[0144] Residual adjustment method 1

[0145] The following assumes that the first residual image includes a first residual value, and the second residual image includes a second residual value corresponding to the first residual value (that is, the second residual value is a residual value obtained after adjusting the first residual value). The following is introduced using an example of adjusting the residual value of the first residual image using a fixed first offset value.

[0146] For example, since the residual values ​​in a residual image can be either positive or negative, the first offset value can be adjusted so that the first and second residual values ​​satisfy at least one of the following conditions: if the first residual value is positive, the second residual value is equal to the difference between the first residual value and the first offset value; if the first residual value is negative, the second residual value is equal to the sum of the first residual value and the first offset value; if the first residual value is 0, no adjustment is performed. After this adjustment, the overall absolute residual value of the first residual image will decrease. Testing has found that in most cases, the residual values ​​in the first residual image are larger than the actual residual values. Therefore, reducing the residual values ​​of the first residual image helps improve the accuracy of the reconstructed image. Taking Figures 5A and 5B as examples, Figure 5A shows an example of the first residual image, and Figure 5B shows an example of the second residual image. The first offset value can be set to 1 to adjust the residual value at each pixel position in Figure 5A. For residual values ​​greater than 0, an offset value of 1 is subtracted; for residual values ​​less than 0, an offset value of 1 is added; for residual values ​​equal to 0, no adjustment is performed. After adjustment, the residual value shown in FIG5B is obtained. Using a fixed offset value to adjust the residual value is very simple to implement and does not increase coding complexity.

[0147] As an example, a pseudo code for performing residual adjustment on the first residual image may be as follows:

[0148] if(RecAfterALF–RecBeforeALF>0)

[0149] Rec_new = RecAfterALF - 1;

[0150] else if(RecAfterALF–RecBeforeALF<0)

[0151] Rec_new = RecAfterALF + 1;

[0152] else

[0153] Rec_new = RecAfterALF;

[0154] For another example, the adjustment of the first offset value can make the first residual value and the second residual value satisfy at least one of the following: if the first residual value is positive, the second residual value is equal to the sum of the first residual value and the first offset value; if the first residual value is negative, the second residual value is equal to the difference between the first residual value and the first offset value.

[0155] For another example, the adjustment of the first offset value can make the first residual value and the second residual value satisfy: if the first residual value is not 0, the second residual value is equal to the sum of the first residual value and the first offset value.

[0156] For another example, the adjustment of the first offset value can make the first residual value and the second residual value satisfy: if the first residual value is not 0, the second residual value is equal to the difference between the first residual value and the first offset value.

[0157] Residual adjustment method 2

[0158] Adjust the residual values within a specific value range in the first residual image.

[0159] For example, the first residual image includes a third residual value. If the third residual value belongs to at least one value range, perform residual adjustment on the third residual value; if the third residual value does not belong to at least one value range, do not perform residual adjustment on the third residual value.

[0160] As an example, perform local residual adjustment within the value range (minimum value min to maximum value max), where both min and max are positive numbers. For a positive residual value res, adjust it only when (min < res < max); for a negative residual value res, adjust it only when (-max < res < -min).

[0161] Residual adjustment method 3

[0162] The residuals in different value ranges in the first residual image are adjusted using different offset values.

[0163] The multiple offset values can be understood as multiple offset values with different precisions. For larger residual values, use a lower-precision offset value (larger value) to adjust them; for smaller residual values, use a higher-precision offset value (smaller value) to adjust them.

[0164] As an example, assuming that for the current pixel of the current slice, the corresponding residual value is res, and the offset value corresponding to res is RA_FACTOR, the value of RA_FACTOR can be derived using the following strategy (expressed in pseudo code):

[0165] Among them, {x1, x2, x3} represents a positive residual value, {y1, y2, y3} represents a negative residual value, and {a1, a2, a3} and {b1, b2, b3} are both preset candidate offset values.

[0166] It should be understood that the above-mentioned various residual adjustment methods can be used individually or in combination. The offset value in the various residual adjustment methods can be a single offset value or a set of offset values ​​formed by multiple offset values ​​with different values. The multiple offset values ​​can be used to adjust the residual values ​​within multiple value ranges.

[0167] In step S650 , a third reconstructed image is determined according to the first reconstructed image and the second residual image.

[0168] In some implementations, performing step S650 may include taking the sum of the first reconstructed image and the second residual image as the third reconstructed image. For example, the sum of the values ​​at corresponding pixel positions in the first reconstructed image and the second residual image may be taken as the pixel value of the third reconstructed image.

[0169] As can be seen from the above description, the embodiment of the present application does not directly use the residual information determined by loop filtering (corresponding to the first residual image described above), but instead adjusts the residual information output by loop filtering before use (i.e., uses the second residual image). Adjusting the residual information may correct errors caused by inconsistencies between the training and actual testing processes, thereby improving encoding performance.

[0170] The above describes in detail how to adjust the residual image determined based on loop filtering. The following describes in detail the relevant parameters (such as syntax elements) that may be involved in the residual image adjustment process.

[0171] In some implementations, first identification information may be written into the bitstream. The first identification information may be used to indicate whether residual adjustment is performed on the first residual image (or whether residual adjustment is allowed on the first residual image). The first identification information may include a first value and a second value. The first value may be, for example, 1 or true. The second value may be, for example, 0 or false. If the value of the first identification information is the first value, the encoder is instructed to perform residual adjustment on the first residual image; if the value of the first identification information is 0, the encoder may be instructed not to perform residual adjustment on the first residual image.

[0172] The first identification information may refer to identification information at a slice level; or, the first identification information may refer to identification information at a coding tree unit (CTU) level; or, the first identification information may refer to identification information at a coding unit (CU) level.

[0173] For example, if the reconstructed image belongs to a slice-level luminance component, the first identification information may be set to sh_ra_enabled_flag. The value of sh_ra_enabled_flag may include 0 and 1. If the value of sh_ra_enabled_flag is 1, it indicates that residual adjustment is performed on the reconstructed image; if the value of sh_ra_enabled_flag is 0, it indicates that residual adjustment is not performed on the reconstructed image.

[0174] For another example, if the reconstructed image belongs to the slice-level chroma Cb component, the first identification information can be set to sh_ra_cb_enabled_flag. The value of sh_ra_cb_enabled_flag can include 0 and 1. If the value of sh_ra_cb_enabled_flag is 1, it indicates that residual adjustment is performed on the reconstructed image; if the value of sh_ra_cb_enabled_flag is 0, it indicates that residual adjustment is not performed on the reconstructed image.

[0175] Whether to write the first identification information into the codestream can be determined based on the rate-distortion cost corresponding to the second reconstructed image and the third reconstructed image. The rate-distortion cost corresponding to the second reconstructed image can be determined by comparing the second reconstructed image with the original image. Similarly, the rate-distortion cost corresponding to the third reconstructed image can be determined by comparing the third reconstructed image with the original image. For example, if the rate-distortion cost corresponding to the third reconstructed image is less than the rate-distortion cost corresponding to the second reconstructed image, the value of the first identification information is the first value. For another example, if the rate-distortion cost corresponding to the third reconstructed image is greater than or equal to the rate-distortion cost corresponding to the second reconstructed image, the value of the first identification information is the second value.

[0176] In some implementations, second identification information may be written into the code stream. The second identification information may be used to indicate whether residual adjustment is performed on the coding tree unit (or coding unit) (or whether residual adjustment is allowed on the coding tree unit). The second identification information may include a third value and a fourth value. The third value may be, for example, 1 or true. The fourth value may be, for example, 0 or false. If the value of the second identification information is the third value, it instructs the encoding end to perform residual adjustment on the coding tree unit; if the value of the second identification information is the fourth value, it may instruct the encoding end not to perform residual adjustment on the coding tree unit.

[0177] For example, if the reconstructed image belongs to the block-level luminance component, the second identification information can be set to ra_ctb_enabled_flag[x][y]. The value of ra_ctb_enabled_flag[x][y] can include 1 and 0. If the value of ra_ctb_enabled_flag[x][y] is 1, it indicates that residual adjustment is performed on the coding tree unit; if the value of ra_ctb_enabled_flag[x][y] is 0, it indicates that residual adjustment is not performed on the coding tree unit.

[0178] It should be understood that the second identification information and the first identification information mentioned above can be used in combination. For example, when the value of the second identification information is the third value (i.e., residual adjustment of the coding tree unit is allowed), the first identification information can continue to be encoded. For another example, when the value of the second identification information is the fourth value (i.e., residual adjustment of the coding tree unit is not allowed), the first identification information is not encoded.

[0179] As mentioned above, the residual value can be adjusted based on multiple offset values. Therefore, the first offset value can be determined based on the rate-distortion cost corresponding to multiple candidate offset values. In some implementations, the first index information can be written into the code stream. The first index information is used to indicate the first offset value from multiple candidate offset values. For example, two candidate offset values ​​can be pre-set: 1 and 2. Among them, the value of the first index information corresponding to the candidate offset value 1 is 0, and the value of the first index information corresponding to the candidate offset value 2 is 1. If the value of the first index information is 1, it can be determined that the residual value of the first residual image is adjusted using the offset value 2. For example, if the first residual value in the first residual image is a positive number, the first residual value is subtracted by 2; if the first residual value in the first residual image is a negative number, the first residual value is added by 2; if the first residual value in the first residual image is 0, the first residual value remains unchanged.

[0180] The residual adjustment method used by the codec may include only one residual adjustment method or multiple residual adjustment methods. If only one residual adjustment method is included, when the encoder determines to adjust the first residual image, the residual adjustment method can be directly used for adjustment. If multiple residual adjustment methods are included, the encoder can write index information indicating the first residual adjustment method currently used into the bitstream. Still taking the residual adjustment of the first residual image as an example, the second index information can be written into the bitstream. The second index information can be used to indicate that the first residual image uses the first residual adjustment method among multiple residual adjustment methods. For example, the preset residual adjustment method includes two residual adjustment methods. The index of residual adjustment method one is 0, and the index of residual adjustment method two is 1. If the value of the second index information is 1, it means that residual adjustment method two is used to adjust the residual value of the first residual image.

[0181] The first residual adjustment method ultimately used can be determined based on the rate-distortion costs corresponding to multiple residual adjustment methods. For example, multiple residual images can be determined based on the multiple residual adjustment methods. Then, the first residual adjustment method is determined based on the rate-distortion costs corresponding to the multiple residual images (for example, the first residual adjustment method is the residual adjustment method with the optimal rate-distortion cost). For example, multiple reconstructed images can be first determined based on the multiple residual images, and then the rate-distortion costs corresponding to the multiple residual images can be determined by comparing the multiple reconstructed images with the original images.

[0182] In some implementations, the first index information and / or the second index information may be used in combination with the first identification information mentioned above. For example, if the first identification information indicates that residual adjustment should be performed on the first residual image, the first index information and / or the second index information may be parsed to determine the residual adjustment method. For another example, if the first identification information indicates that residual adjustment should not be performed on the first residual image, the first index information and / or the second index information may not be parsed.

[0183] As mentioned above, if multiple residual adjustment methods are included, the bitstream can carry index information indicating the residual adjustment method currently in use. Alternatively, in some implementations, the residual adjustment method can be determined based on the coding parameters corresponding to the current block. The coding parameters here may include: the frame type corresponding to the current block (e.g., I-frame / P-frame / B-frame), the quantization parameter corresponding to the current block (e.g., QP for general test conditions or QP for slice level), and the filter coefficient set for loop filtering corresponding to the current block (e.g., fixed subset or APS subset).

[0184] The current block mentioned above may be a coding tree unit (CTU) or a coding unit (CU).

[0185] The following examples are used to describe the embodiments of the present application in more detail. It should be noted that the following examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. It is apparent that those skilled in the art can make various equivalent modifications or changes based on the examples given, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0186] This example proposes an output information adjustment method that further optimizes the coding performance of the adaptive loop filter by correcting the output residual information of the adaptive loop filter. Figure 7 shows an example diagram of the residual information adjustment module (RA) located in the codec. RA is the residual information adjustment module proposed in this example, which essentially adjusts the output residual of the adaptive loop filter (ALF / CCALF). As shown in Figure 7, at the encoder end, the input and output information of the CCALF are first obtained and the difference is taken to obtain the output residual information. The residual information is then input into the RA module proposed in this article for adjustment. Finally, the corrected residual is superimposed on the CCALF input information to obtain the residual-adjusted CCALF output result.

[0187] The original CCALF output and the residual-adjusted CCALF output are compared with the rate-distortion cost of the original image to determine which output to use. The selected output is encoded into the bitstream for the decoder to read. At the decoder, after parsing the output actually used by the encoder, the reconstructed image is filtered.

[0188] It should be noted that Figure 7 illustrates adjusting the output of the CCALF. The proposed method can also perform similar adjustments on the output of the ALF, or perform a combined adjustment on the ALF and CCALF. As shown in Figure 8, the filter modules that can be adjusted in the proposed scheme are:

[0189] (1) Brightness ALF;

[0190] (2) Chroma ALF;

[0191] (3)CCALF;

[0192] (4) Combination of chroma ALF and CCALF (the residual is calculated as the difference between the output information of CCALF and the input information of chroma ALF).

[0193] This example is implemented in both the loop filter modules on the encoder and decoder sides. This example proposes a residual information adjustment method that, by making relatively simple adjustments to the output residual information of the adaptive loop filter, further improves encoding performance without substantially increasing encoding and decoding complexity. The specific implementation of this example in the loop filter module on the encoder side is as follows:

[0194] When the encoder enters the loop filter module, it processes according to the specified filter order. When it enters the residual information adjustment module, for the current slice, the luminance component is processed first:

[0195] a) First, determine whether the current slice has enabled the luminance ALF according to the sh_alf_enabled_flag. When sh_alf_enabled_flag is equal to 1, for the luminance component of the current slice, attempt to process it through the residual information adjustment module and jump to b); when sh_alf_enabled_flag is equal to 0, for the luminance component of the current slice, do not attempt to process it through the luminance residual information adjustment module and jump to d);

[0196] b) For the luminance component of the current slice, subtract the reconstructed image before inputting ALF from the filtered reconstructed image output by ALF to obtain a residual image. Apply the adjustment scheme designed in this article (introduced in detail later) to the residual image and then superimpose it on the input reconstructed image to obtain the output image after residual adjustment, and jump to c);

[0197] c) Compare the filtered reconstructed image output by ALF and the filtered output image adjusted in this article with the original image respectively and calculate the rate-distortion cost, denoted as cost Cost(C). Compare the two costs. If C_RA < C_NNLF, then use the filtered image output by the residual information adjustment module as the final filtered image, assign sh_ra_enabled_flag equal to 1, and further transmit the slice-level luminance residual adjustment offset value serial number sh_ra_index into the bitstream; otherwise, if C_RA ≥ C_NNLF, still use the filtered image output by the original ALF as the final filtered image and assign sh_ra_enabled_flag equal to 0. Jump to d);

[0198] d) If the processing of the luminance component of the current slice has been completed, then load the chrominance component of the current slice for processing.

[0199] For the current slice, continue to process the chrominance component (take the Cb component as an example, the processing process of the Cr component is similar):

[0200] a) First, determine whether the current slice has enabled the luminance ALF according to the sh_alf_enabled_flag. When sh_alf_enabled_flag is equal to 1, for the chrominance component of the current slice, attempt to process it through the residual information adjustment module and jump to b); when sh_alf_enabled_flag is equal to 0, for the chrominance component of the current slice, do not attempt to process it through the chrominance residual information adjustment module and jump to e);

[0201] b) First, determine whether the current slice has enabled Cb ALF or Cb CCALF according to the sh_alf_cb_enabled_flag or sh_alf_cc_cb_enabled_flag flag, that is, determine whether this chrominance component has been filtered. When the OR operation equals 1, for the chrominance component Cb of the current slice, attempt to process it through the residual information adjustment module and jump to c); when the OR operation equals 0, for the chrominance component Cb of the current slice, do not attempt to process it through the chrominance residual information adjustment module and jump to e);

[0202] c) For the chrominance component Cb of the current slice, subtract the reconstructed image before input from the filtered reconstructed image output to obtain the residual image. Apply the adjustment scheme designed in this paper (introduced in detail later) to the residual image and add it to the input reconstructed image to obtain the output image after residual adjustment, then jump to d);

[0203] d) Compare the existing filtered output image and the filtered output image adjusted in this paper with the original image respectively and calculate the rate - distortion cost, denoted as cost Cost(C). Compare the two costs. If C_RA < C_NNLF, then use the filtered image output by the residual information adjustment module as the final filtered image, assign sh_ra_cb_enabled_flag equal to 1, and further transmit the slice - level chrominance Cb residual adjustment offset value serial number sh_ra_cb_index into the bitstream; otherwise, if C_RA ≥ C_NNLF, still use the original filtered output image as the final filtered image and assign sh_ra_cb_enabled_flag equal to 0. Then jump to d);

[0204] e) If all color components of the current slice have been processed, end.

[0205] Definition of slice header:

[0206] a) As shown in the above table, if the slice - level luminance ALF enable flag sh_alf_enabled_flag is 1, define the semantics: slice - level luminance residual adjustment enable flag sh_ra_enabled_flag; if sh_ra_enabled_flag is 1, define the semantics: slice - level luminance residual adjustment offset value serial number sh_ra_index.

[0207] b) As shown in the above table, if the slice-level chroma Cb ALF enable flag sh_alf_cb_enabled_flag or the slice-level chroma Cb CCALF enable flag sh_alf_cc_cb_enabled_flag is 1, the semantics are defined: the slice-level chroma Cb residual adjustment enable flag sh_ra_cb_enabled_flag; if sh_ra_cb_enabled_flag is 1, the semantics are defined: the slice-level chroma Cb residual adjustment offset value index sh_ra_cb_index.

[0208] c) As shown in the above table, if the slice-level chroma Cr ALF enable flag sh_alf_cr_enabled_flag or the slice-level chroma Cr CCALF enable flag sh_alf_cc_cr_enabled_flag is 1, the semantics are defined: slice-level chroma Cr residual adjustment enable flag sh_ra_cr_enabled_flag; if sh_ra_cr_enabled_flag is 1, the semantics are defined: slice-level chroma Cr residual adjustment offset value index sh_ra_cr_index.

[0209] As mentioned above, the purpose of this example is to further improve the coding performance by adjusting the output residual information of the adaptive loop filter.

[0210] The basic goal of this example is to minimize the output residual of the adaptive loop filter. Specifically, for a residual image, the residual value at each pixel can be positive or negative. To minimize the residual, this example subtracts a fixed value (positive) from the positive residual value and adds the fixed value to the negative residual value. No adjustment is made for residual values ​​equal to 0, thus minimizing the overall absolute residual value.

[0211] As shown in Figure 5, assuming a fixed value of (+1), Figure 5A shows the residual image of the original ALF output, and Figure 5B shows the adjusted residual image. Finally, the residual image is superimposed on the input reconstructed image to obtain the filtered reconstructed image.

[0212] This example can set multiple offset values ​​for the residual information adjustment method, select one of the values ​​through rate-distortion cost calculation, and encode its index into the bitstream for the decoder to read and process.

[0213] When the solution in this article is enabled, the pseudo code of an adjustment example is as follows:

[0214] if(RecAfterALF–RecBeforeALF>0)

[0215] Rec_new = RecAfterALF - 1;

[0216] else if (RecAfterALF – RecBeforeALF < 0)

[0217] Rec_new = RecAfterALF + 1;

[0218] else

[0219] Rec_new = RecAfterALF;

[0220] Next, several implementation manners in this example will be introduced in detail.

[0221] Implementation manner 1:

[0222] In the current output information adjustment scheme, at the slice level, for positive (>0) residual values, a certain fixed value is uniformly subtracted; for negative (<0) residual values, a certain fixed value is uniformly added. Subsequently, further improvements can also be made according to the residual characteristics, and adjustments can be made based on smaller coding units (such as CTUs) or other-sized regions. The specific syntax design form can be as follows (taking the luminance component as an example, the chrominance component can also adopt a similar syntax definition):

[0223] For each coding unit such as a CTU, when the slice-level luminance ALF enable flag sh_alf_enabled_flag is 1, the following semantics are defined: block-level luminance residual adjustment enable flag ra_ctb_enabled_flag[x][y].

[0224] When ra_ctb_enabled_flag[x][y] is 1, the following semantics are defined: block-level luminance residual adjustment offset value serial number ra_ctb_index[x][y].

[0225] Coding tree unit definition:

[0226] Implementation manner 2:

[0227] In the current output information adjustment scheme, at the slice level, for positive (>0) residual values, a certain fixed value is uniformly subtracted; for negative (<0) residual values, a certain fixed value is uniformly added. Subsequently, further restrictions can be made according to the residual characteristics, that is, not only compared with the value 0. For example, local residual adjustment can be considered within the interval (minimum value min to maximum value max), where both min and max are positive numbers. Then, for a positive residual value res, adjustment is only made when (min < res < max); for a negative residual value res, adjustment is only made when (-max < res < -min).

[0228] Implementation manner 3:

[0229] The current output information adjustment scheme is based on the slice level. For positive (>0) residual values, a fixed value is uniformly subtracted; for negative (<0) residual values, a fixed value is uniformly added. Further strategy changes can be made based on the residual characteristics. For example, four strategies can be designed:

[0230] Strategy 1 (this paper's solution): For positive (>0) residual values, subtract a fixed value; for negative (<0) residual values, add a fixed value;

[0231] Strategy 2: For positive (>0) residual values, add a fixed value; for negative (<0) residual values, subtract a fixed value;

[0232] Strategy 3: For non-zero residual values, add a fixed value;

[0233] Strategy 4: For non-zero residual values, subtract a fixed value uniformly;

[0234] The encoder can try out the above four strategies, determine the optimal strategy through rate-distortion optimization, and transmit its sequence number to the decoder for reading and use.

[0235] Implementation 4:

[0236] The current output information adjustment scheme uses a fixed-value adjustment operation on the pixel residual information at the slice level. For example, a fixed value is subtracted from all positive residual values. Alternatively, the residual values ​​can be segmented based on their characteristics, and fixed-value adjustments of different scales can be attempted. For example, a larger fixed value is set for larger residual values, and a smaller fixed value is set for smaller residual values.

[0237] A simple example and pseudocode are as follows:

[0238] Assume that for the current pixel point of the current slice, the corresponding residual value is res, and the fixed value to be determined is RA_FACTOR. The specific fixed value derivation strategy is as follows.

[0239] Among them, {x1, x2, x3} represents a positive residual value, {y1, y2, y3} represents a negative residual value, and {a1, a2, a3} and {b1, b2, b3} are preset candidate fixed values.

[0240] Implementation 5:

[0241] The current output information adjustment scheme adjusts the output information of the adaptive loop filter (ALF / CCALF). This scheme can be subsequently migrated to other loop filter modules, such as the luminance mapping and chroma scaling (LMCS), deblocking filter (DBF), sample offset compensation (SAO), adaptive loop filter (ALF), and cross-component adaptive loop filter (CCALF) filtering tools shown in Figure 2. The scheme in this article can be applied to these tools alone or in combination. By obtaining the input and output reconstructed images and subtracting them to obtain the residual image, the output residual information can be adjusted.

[0242] Implementation 6:

[0243] The current output information adjustment scheme does not perform feature recognition for different coding blocks in the current residual image; instead, it performs addition and subtraction operations based on the positive and negative values ​​of pixel values. In practice, different coding blocks may require different adjustment methods. This scheme could be further improved to implement an output information adjustment method based on coding characteristics.

[0244] Optional coding features include: the frame type (I / P / B) of the current coding block, quantization parameters (baseQP / sliceQP), etc., as well as coding features related to the adaptive loop filter, such as the filter coefficient set (fixed subset / APS subset) selected for the current coding block.

[0245] The specific implementation method is shown in Figure 9. For the current residual image, it is divided based on the coding block size (such as CTU), and the coding characteristics of the current block (such as I / P / B) are analyzed. Then, different residual adjustment methods are used for different I / P / B blocks.

[0246] The method embodiment of the present application is described in detail above in conjunction with Figures 1 to 9 . The device embodiment of the present application is described in detail below in conjunction with Figures 10 to 13 . It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for portions not described in detail, reference can be made to the above method embodiment.

[0247] FIG10 is a schematic diagram of the structure of a decoder provided by an embodiment of the present application. As shown in FIG10 , the decoder 1000 includes: a first determination unit 1010 , a second determination unit 1020 , a third determination unit 1030 , a fourth determination unit 1040 , and a fifth determination unit 1050 .

[0248] The first determining unit 1010 is configured to parse the code stream and determine a first reconstructed image.

[0249] The second determining unit 1020 is configured to perform a loop filtering operation on the first reconstructed image to determine a second reconstructed image.

[0250] The third determining unit 1030 is configured to determine a first residual image according to the first reconstructed image and the second reconstructed image.

[0251] The fourth determining unit 1040 is configured to perform residual adjustment on the first residual image to determine a second residual image.

[0252] The fifth determining unit 1050 is configured to determine a third reconstructed image according to the first reconstructed image and the second residual image.

[0253] In some implementations, the decoder 1000 includes a sixth determination unit 1060 configured to parse the code stream and determine first identification information, where the first identification information is used to indicate whether to perform residual adjustment; the third determination unit 1030 is configured to determine the first residual image based on the first reconstructed image and the second reconstructed image if the first identification information indicates to perform residual adjustment.

[0254] In some implementations, the first identification information is slice-level identification information; or, the first identification information is coding tree unit-level identification information; or, the first identification information is coding unit-level identification information.

[0255] In some implementations, the decoder 1000 includes a seventh determining unit 1070 configured to parse the code stream and determine first index information, where the first index information is used to indicate a first offset value from a plurality of candidate offset values.

[0256] In some implementations, the decoder 1000 includes an eighth determining unit 1080 configured to parse the bitstream and determine second index information, where the second index information is used to indicate a first residual adjustment method from among multiple residual adjustment methods.

[0257] In some implementations, different residual adjustment methods among the multiple residual adjustment methods process positive residual values ​​and / or negative residual values ​​in the first residual image in different ways.

[0258] In some implementations, the first residual image includes a first residual value, the second residual image includes a second residual value corresponding to the first residual value, and the first residual value and the second residual value satisfy at least one of the following: if the first residual value is a positive number, the second residual value is equal to the difference between the first residual value and the first offset value; if the first residual value is a negative number, the second residual value is equal to the sum of the first residual value and the first offset value; if the first residual value is 0, the second residual value is equal to the first residual value.

[0259] In some implementations, the fourth determining unit 1040 is configured to perform residual adjustment on the first residual image according to a plurality of offset values, where the plurality of offset values ​​are respectively used to adjust residual values ​​within a plurality of value ranges.

[0260] In some implementations, the fourth determination unit 1040 is configured to perform residual adjustment on the third residual value in the first residual image if the third residual value belongs to at least one value range; and / or not perform residual adjustment on the third residual value if the third residual value in the first residual image does not belong to the at least one value range.

[0261] In some implementations, a residual adjustment method for the current block in the first residual image is determined based on encoding parameters corresponding to the current block.

[0262] In some implementations, the encoding parameter includes at least one of the following: a frame type corresponding to the current block; a quantization parameter corresponding to the current block; and a filter coefficient set for loop filtering corresponding to the current block.

[0263] In some implementations, the current block is a coding tree unit or a coding unit.

[0264] In some implementations, the loop filtering operation includes at least one of: luma mapping and chroma scaling, deblocking filtering, sample offset compensation, adaptive loop filtering, and cross-component adaptive loop filtering.

[0265] In some implementations, the first residual image is determined based on a difference between the first reconstructed image and the second reconstructed image.

[0266] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0267] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0268] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the decoder 1000. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the decoding method in the first embodiment.

[0269] Based on the composition of the above-mentioned decoder 1000 and the computer-readable storage medium, refer to Figure 11, which shows a specific hardware structure diagram of the decoder 1100 provided in an embodiment of the present application. As shown in Figure 11, the decoder 1100 may include: a communication interface 1110, a memory 1120 and a processor 1130; each component is coupled together through a bus system 1140. It can be understood that the bus system 1140 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1140 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 1140 in Figure 11. Among them,

[0270] The communication interface 1110 is used to receive and send signals when sending and receiving information with other external network elements.

[0271] The memory 1120 is used to store computer programs.

[0272] Processor 1130 is configured to, when running the computer program, execute the following: parsing a code stream to determine a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image based on the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; and determining a third reconstructed image based on the first reconstructed image and the second residual image.

[0273] It is understood that the memory 1120 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1120 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0274] The processor 1130 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1130. The above-mentioned processor 1130 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1120 , and the processor 1130 reads the information in the memory 1120 and completes the steps of the above method in combination with its hardware.

[0275] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0276] Optionally, as another embodiment, the processor 1130 is further configured to execute the decoding method described in the above embodiment when running the computer program.

[0277] FIG12 is a schematic diagram of the structure of an encoder provided by an embodiment of the present application. As shown in FIG12 , the encoder 1200 includes: a first determining unit 1210 , a second determining unit 1220 , a third determining unit 1230 , a fourth determining unit 1240 , and a fifth determining unit 1250 .

[0278] The first determining unit 1210 is configured to determine a first reconstructed image.

[0279] The second determining unit 1220 is configured to perform a loop filtering operation on the first reconstructed image to determine a second reconstructed image.

[0280] The third determining unit 1230 is configured to determine a first residual image according to the first reconstructed image and the second reconstructed image.

[0281] The fourth determining unit 1240 is configured to perform residual adjustment on the first residual image to determine a second residual image.

[0282] The fifth determining unit 1250 is configured to determine a third reconstructed image according to the first reconstructed image and the second residual image.

[0283] In some implementations, the encoder 1200 includes: a first encoding unit 1260 configured to write first identification information into a bitstream, where the first identification information is used to indicate whether to perform residual adjustment.

[0284] In some implementations, if the rate-distortion cost corresponding to the third reconstructed image is less than the rate-distortion cost corresponding to the second reconstructed image, the first identification information indicates that residual adjustment is to be performed; and / or, if the rate-distortion cost corresponding to the third reconstructed image is greater than or equal to the rate-distortion cost corresponding to the second reconstructed image, the first identification information indicates that no residual adjustment is to be performed.

[0285] In some implementations, the first identification information is slice-level identification information; or, the first identification information is coding tree unit-level identification information; or, the first identification information is coding unit-level identification information.

[0286] In some implementations, the encoder 1200 includes: a second encoding unit 1270 configured to write first index information into a bitstream, where the first index information is used to indicate a first offset value from a plurality of candidate offset values.

[0287] In some implementations, the first offset value is determined based on rate-distortion costs corresponding to the multiple candidate offset values.

[0288] In some implementations, the encoder 1200 includes: a third encoding unit 1280 configured to write second index information into a bitstream, where the second index information is used to indicate a first residual adjustment method from among multiple residual adjustment methods.

[0289] In some implementations, different residual adjustment methods among the multiple residual adjustment methods process positive residual values ​​and / or negative residual values ​​in the first residual image in different ways.

[0290] In some implementations, the first residual adjustment method is determined based on rate-distortion costs corresponding to the multiple residual adjustment methods.

[0291] In some implementations, the first residual image includes a first residual value, the second residual image includes a second residual value corresponding to the first residual value, and the first residual value and the second residual value satisfy at least one of the following: if the first residual value is a positive number, the second residual value is equal to the difference between the first residual value and the first offset value; if the first residual value is a negative number, the second residual value is equal to the sum of the first residual value and the first offset value; if the first residual value is 0, the second residual value is equal to the first residual value.

[0292] In some implementations, the fourth determining unit 1240 is configured to perform residual adjustment on the first residual image according to a plurality of offset values, where the plurality of offset values ​​are respectively used to adjust residual values ​​within a plurality of value ranges.

[0293] In some implementations, the fourth determination unit 1240 is configured to perform residual adjustment on the third residual value in the first residual image if the third residual value belongs to at least one value range; and / or not perform residual adjustment on the third residual value if the third residual value in the first residual image does not belong to the at least one value range.

[0294] In some implementations, a residual adjustment method for the current block in the first residual image is determined based on encoding parameters corresponding to the current block.

[0295] In some implementations, the encoding parameter includes at least one of the following: a frame type corresponding to the current block; a quantization parameter corresponding to the current block; and a filter coefficient set for loop filtering corresponding to the current block.

[0296] In some implementations, the current block is a coding tree unit or a coding unit.

[0297] In some implementations, the loop filtering operation includes at least one of: luma mapping and chroma scaling, deblocking filtering, sample offset compensation, adaptive loop filtering, and cross-component adaptive loop filtering.

[0298] In some implementations, the first residual image is determined based on a difference between the first reconstructed image and the second reconstructed image.

[0299] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0300] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0301] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 1200. The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the decoding method in the first embodiment.

[0302] Based on the composition of the above-mentioned encoder 1200 and the computer-readable storage medium, refer to Figure 13, which shows a specific hardware structure diagram of the encoder 1300 provided in an embodiment of the present application. As shown in Figure 13, the encoder 1300 may include: a communication interface 1310, a memory 1320 and a processor 1330; each component is coupled together through a bus system 1340. It can be understood that the bus system 1340 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1340 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 1340 in Figure 13. Among them,

[0303] The communication interface 1310 is used to receive and send signals when sending and receiving information with other external network elements.

[0304] The memory 1320 is used to store computer programs.

[0305] Processor 1330 is configured to, when running the computer program, execute the following: determining a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image based on the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; and determining a third reconstructed image based on the first reconstructed image and the second residual image.

[0306] It is understood that the memory 1320 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1320 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0307] The processor 1330 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1330. The above-mentioned processor 1330 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1320 , and the processor 1330 reads the information in the memory 1320 and completes the steps of the above method in combination with its hardware.

[0308] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0309] Optionally, as another embodiment, the processor 1330 is further configured to execute the encoding method described in the above embodiment when running the computer program.

[0310] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0311] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0312] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0313] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0314] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0315] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A decoding method, applied to a decoder, comprising: Parsing the code stream to determine the first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image according to the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; A third reconstructed image is determined according to the first reconstructed image and the second residual image.

2. The method according to claim 1, wherein The method further comprises: Parsing the bitstream to determine first identification information, where the first identification information is used to indicate whether to perform residual adjustment; The determining a first residual image according to the first reconstructed image and the second reconstructed image includes: If the first identification information indicates to perform residual adjustment, the first residual image is determined according to the first reconstructed image and the second reconstructed image.

3. The method according to claim 2, wherein: The first identification information is slice-level identification information; or, The first identification information is identification information at the coding tree unit level; or, The first identification information is identification information at the coding unit level.

4. The method according to claim 1, wherein The method further comprises: Parse the code stream to determine first index information, where the first index information is used to indicate a first offset value from a plurality of candidate offset values.

5. The method according to claim 1, wherein The method further comprises: The code stream is parsed to determine second index information, where the second index information is used to indicate a first residual adjustment method from among a plurality of residual adjustment methods.

6. The method according to claim 5, wherein: Different residual adjustment methods among the multiple residual adjustment methods process positive residual values and / or negative residual values in the first residual image in different ways.

7. The method according to claim 1, wherein The first residual image includes a first residual value, the second residual image includes a second residual value corresponding to the first residual value, and the first residual value and the second residual value satisfy at least one of the following: If the first residual value is a positive number, the second residual value is equal to the difference between the first residual value and the first offset value; If the first residual value is a negative number, the second residual value is equal to the sum of the first residual value and the first offset value; If the first residual value is 0, the second residual value is equal to the first residual value.

8. The method according to claim 1, wherein The performing residual adjustment on the first residual image includes: Residual adjustment is performed on the first residual image according to a plurality of offset values, where the plurality of offset values are respectively used to adjust residual values within a plurality of value ranges.

9. The method according to claim 1, wherein The performing residual adjustment on the first residual image includes: If the third residual value in the first residual image belongs to at least one value range, performing residual adjustment on the third residual value; and / or, If the third residual value in the first residual image does not belong to the at least one value range, no residual adjustment is performed on the third residual value.

10. The method according to claim 1, wherein The residual adjustment method of the current block in the first residual image is determined based on the encoding parameters corresponding to the current block.

11. The method according to claim 10, wherein: The encoding parameters include at least one of the following: The frame type corresponding to the current block; The quantization parameter corresponding to the current block; The filter coefficient set for loop filtering corresponding to the current block.

12. The method according to claim 10 or 11, wherein: The current block is a coding tree unit or a coding unit.

13. The method according to claim 1, wherein The loop filtering operation includes at least one of the following: luma mapping and chroma scaling, deblocking filtering, sample offset compensation, adaptive loop filtering, and cross-component adaptive loop filtering.

14. The method according to claim 1, wherein The first residual image is determined based on a difference between the first reconstructed image and the second reconstructed image.

15. A coding method, applied to an encoder, comprising: determining a first reconstructed image; performing a loop filtering operation on the first reconstructed image to determine a second reconstructed image; determining a first residual image according to the first reconstructed image and the second reconstructed image; performing residual adjustment on the first residual image to determine a second residual image; A third reconstructed image is determined according to the first reconstructed image and the second residual image.

16. The method according to claim 15, wherein The method further comprises: The first identification information is written into the bitstream, where the first identification information is used to indicate whether to perform residual adjustment.

17. The method according to claim 16, wherein: If the rate-distortion cost corresponding to the third reconstructed image is less than the rate-distortion cost corresponding to the second reconstructed image, the first identification information indicates to perform residual adjustment; and / or, If the rate-distortion cost corresponding to the third reconstructed image is greater than or equal to the rate-distortion cost corresponding to the second reconstructed image, the first identification information indicates that residual adjustment is not performed.

18. The method according to claim 16 or 17, wherein: The first identification information is slice-level identification information; or, The first identification information is identification information at the coding tree unit level; or, The first identification information is identification information at the coding unit level.

19. The method according to claim 15, wherein The method further comprises: First index information is written into a code stream, where the first index information is used to indicate a first offset value from a plurality of candidate offset values.

20. The method according to claim 19, wherein The first offset value is determined based on rate-distortion costs corresponding to the multiple candidate offset values.

21. The method according to claim 15, wherein The method further comprises: Second index information is written into the bitstream, where the second index information is used to indicate a first residual adjustment method from among multiple residual adjustment methods.

22. The method according to claim 21, wherein Different residual adjustment methods among the multiple residual adjustment methods process positive residual values and / or negative residual values in the first residual image in different ways.

23. The method according to claim 21 or 22, wherein The first residual adjustment method is determined based on rate-distortion costs corresponding to the multiple residual adjustment methods.

24. The method according to claim 15, wherein The first residual image includes a first residual value, the second residual image includes a second residual value corresponding to the first residual value, and the first residual value and the second residual value satisfy at least one of the following: If the first residual value is a positive number, the second residual value is equal to the difference between the first residual value and the first offset value; If the first residual value is a negative number, the second residual value is equal to the sum of the first residual value and the first offset value; If the first residual value is 0, the second residual value is equal to the first residual value.

25. The method according to claim 15, wherein The performing residual adjustment on the first residual image includes: Residual adjustment is performed on the first residual image according to a plurality of offset values, where the plurality of offset values are respectively used to adjust residual values within a plurality of value ranges.

26. The method according to claim 15, wherein The performing residual adjustment on the first residual image includes: If the third residual value in the first residual image belongs to at least one value range, performing residual adjustment on the third residual value; and / or, If the third residual value in the first residual image does not belong to the at least one value range, no residual adjustment is performed on the third residual value.

27. The method according to claim 15, wherein The residual adjustment method of the current block in the first residual image is determined based on the encoding parameters corresponding to the current block.

28. The method according to claim 27, wherein The encoding parameters include at least one of the following: The frame type corresponding to the current block; The quantization parameter corresponding to the current block; The filter coefficient set for loop filtering corresponding to the current block.

29. The method according to claim 27 or 28, wherein The current block is a coding tree unit or a coding unit.

30. The method of claim 15, wherein: The loop filtering operation includes at least one of the following: luma mapping and chroma scaling, deblocking filtering, sample offset compensation, adaptive loop filtering, and cross-component adaptive loop filtering.

31. The method of claim 15, wherein: The first residual image is determined based on a difference between the first reconstructed image and the second reconstructed image.

32. A decoder comprising: a first determining unit configured to parse the code stream and determine a first reconstructed image; a second determining unit configured to perform a loop filtering operation on the first reconstructed image to determine a second reconstructed image; a third determining unit, configured to determine a first residual image according to the first reconstructed image and the second reconstructed image; a fourth determining unit configured to perform residual adjustment on the first residual image to determine a second residual image; The fifth determining unit is configured to determine a third reconstructed image according to the first reconstructed image and the second residual image.

33. A decoder comprising: memory for storing computer programs; A processor, configured to execute the method according to any one of claims 1 to 14 when running the computer program.

34. An encoder comprising: a first determining unit, configured to determine a first reconstructed image; a second determining unit configured to perform a loop filtering operation on the first reconstructed image to determine a second reconstructed image; a third determining unit, configured to determine a first residual image according to the first reconstructed image and the second reconstructed image; a fourth determining unit configured to perform residual adjustment on the first residual image to determine a second residual image; The fifth determining unit is configured to determine a third reconstructed image according to the first reconstructed image and the second residual image.

35. An encoder comprising: memory for storing computer programs; A processor, configured to perform the method according to any one of claims 15 to 31 when running the computer program.

36. A non-volatile computer-readable storage medium storing a bit stream, wherein the bit stream is generated by an encoding method using an encoder, or the bit stream is decoded by a decoding method using a decoder, wherein: The decoding method is the method according to any one of claims 1 to 14, and the encoding method is the method according to any one of claims 15 to 31.

37. A code stream, comprising a code stream generated by the method according to any one of claims 15 to 31.

38. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 14 or 15 to 31 is implemented.

Images