Coding and decoding method, codec and storage medium

Abstract

The embodiment of the invention provides a coding and decoding method, a codec and a storage medium. The decoding method comprises the following steps: analyzing a code stream, and determining first information; determining a target value of a first scaling factor from a plurality of candidate values of the first scaling factor according to the first information; performing loop filtering on the reconstructed image according to a neural network to determine a first residual image; and correcting the first residual image according to the target value of the first telescopic factor, and determining a reconstructed image after loop filtering.

Description

Coding and decoding method, codec and storage medium Technical Field

[0001] The present application relates to the technical field of video coding and decoding, and in particular to a coding and decoding method, a codec, and a storage medium. Background Art

[0002] A neural network based loop filter (NNLF) can correct the predicted residual image of the neural network based on the derived scaling factor, and determine a reconstructed image after loop filtering based on the corrected residual image.

[0003] However, the derivation method of the scaling factor provided by the related art results in inaccurate values ​​of the scaling factor, thereby reducing the quality of loop filtering.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, a codec, and a storage medium to improve the quality of loop filtering. The following describes various aspects of the present application.

[0006] In a first aspect, a decoding method is provided, applied to a decoder, comprising: parsing a bitstream to determine first information; determining a target value of the first scaling factor from multiple candidate values ​​of the first scaling factor based on the first information; performing loop filtering on a reconstructed image based on a neural network to determine a first residual image; and correcting the first residual image based on the target value of the first scaling factor to determine a reconstructed image after loop filtering.

[0007] In a second aspect, an encoding method is provided, which is applied to an encoder, including: determining a reconstructed image of an original image; performing loop filtering on the reconstructed image according to a neural network to determine a first residual image; correcting the first residual image according to multiple candidate values ​​of a first scaling factor to determine multiple output images; and determining a target value of the first scaling factor from the multiple candidate values ​​based on the multiple output images.

[0008] In a third aspect, a decoder is provided, comprising: a parsing unit configured to parse a bitstream and determine first information; a filtering unit configured to determine a target value of the first scaling factor from multiple candidate values ​​of the first scaling factor based on the first information; perform loop filtering on a reconstructed image based on a neural network to determine a first residual image; and correct the first residual image based on the target value of the first scaling factor to determine a reconstructed image after loop filtering.

[0009] According to a fourth aspect, a decoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method according to the first aspect when running the computer program.

[0010] In a fifth aspect, an encoder is provided, comprising: a determination unit configured to determine a reconstructed image of an original image; a filtering unit configured to perform loop filtering on the reconstructed image according to a neural network to determine a first residual image; correcting the first residual image according to multiple candidate values ​​of a first scaling factor to determine multiple output images; and determining a target value of the first scaling factor from the multiple candidate values ​​based on the multiple output images.

[0011] In a sixth aspect, an encoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method described in the second aspect when running the computer program.

[0012] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the method as described in the first aspect or the second aspect is implemented.

[0013] In an eighth aspect, a computer program product is provided, comprising a computer program, which implements the method described in the first aspect or the second aspect when executed.

[0014] In the embodiment of the present application, a target value of the scaling factor is determined from among the multiple candidate values ​​based on the correction results of the multiple candidate values ​​of the scaling factor. This makes the scaling factor more accurate and helps improve the quality of loop filtering. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a structural diagram illustrating an example of a video encoder to which an embodiment of the present application may be applied.

[0016] FIG2 is a diagram showing an example structure of a video decoder to which an embodiment of the present application can be applied.

[0017] FIG3 is a structural example diagram of a high-complexity neural network filter.

[0018] FIG4 is a diagram showing an example of the structure of a low-complexity neural network filter.

[0019] FIG5 is a schematic diagram of the filtering method of NNLF.

[0020] FIG6 is a schematic diagram showing the relationship between the residual of the original image and the reconstructed image and the prediction residual output by the NNLF.

[0021] FIG7 is a flow chart of a decoding method provided in an embodiment of the present application.

[0022] FIG8 is a flow chart of the encoding method provided in an embodiment of the present application.

[0023] FIG9 is an example diagram of a decoding framework provided in an embodiment of the present application.

[0024] FIG10 is a schematic diagram of the structure of a decoder provided in one embodiment of the present application.

[0025] FIG11 is a schematic diagram of the structure of a decoder provided in another embodiment of the present application.

[0026] FIG12 is a schematic diagram of the structure of an encoder provided in one embodiment of the present application.

[0027] FIG13 is a schematic diagram of the structure of an encoder provided in another embodiment of the present application. DETAILED DESCRIPTION

[0028] FIG1 is a schematic block diagram of a video encoder according to an embodiment of the present application.

[0029] It should be understood that the video encoder 100 can be used to perform lossy compression or lossless compression on an image. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0030] The video encoder 100 can be applied to image data in a luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0031] For example, the video encoder 100 reads video data, and for each image in the video data, divides the image into a number of coding tree units (CTUs). In some examples, CTUs may be referred to as "tree blocks", "largest coding units" (LCUs) or "coding tree blocks" (CTBs). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chroma sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding. A CU may be a rectangular block or a square block. A CU can be further divided into prediction units (PUs) and transform units (TUs), allowing for separation of coding, prediction, and transform, and greater flexibility in processing. In one example, a CTU is divided into CUs using a quadtree, and a CU is divided into TUs and PUs using a quadtree.

[0032] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0033] In some embodiments, as shown in FIG1 , the video encoder 100 may include a prediction unit 110, a residual unit 120, a transform / quantization unit 130, an inverse transform / quantization unit 140, a reconstruction unit 150, a loop filter unit 160, a decoded image buffer 170, and an entropy coding unit 180. It should be noted that the video encoder 100 may include more, fewer, or different functional components.

[0034] Optionally, in this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstruction block.

[0035] In some embodiments, the prediction unit 110 includes an inter-frame prediction unit 111 and an intra-frame prediction unit 112. Because there is a strong correlation between adjacent pixels in a video image, intra-frame prediction is used in video coding and decoding technologies to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent images in a video, inter-frame prediction is used in video coding and decoding technologies to eliminate temporal redundancy between adjacent images, thereby improving coding efficiency.

[0036] The inter-frame prediction unit 111 can be used for inter-frame prediction. Inter-frame prediction can include motion estimation and motion compensation. It can refer to image information from different images. Inter-frame prediction uses motion information to find a reference block from the reference image and generate a prediction block based on the reference block to eliminate temporal redundancy. Inter-frame prediction uses motion information to find a reference block from the reference image and generate a prediction block based on the reference block. Motion information includes the reference image list in which the reference image is located, the reference image index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference image. Here, the integer pixel or fractional pixel block in the reference image found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Reprocessing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.

[0037] The intra-frame prediction unit 112 only refers to information of the same image to predict pixel information within the current code image block to eliminate spatial redundancy.

[0038] Intra-frame prediction has multiple prediction modes. For example, the H-series international digital video coding standard H.264 / AVC has eight angular prediction modes and one non-angular prediction mode. H.265 / HEVC expands this to 33 angular prediction modes and two non-angular prediction modes. HEVC uses planar, DC, and 33 angular modes for a total of 35 intra-frame prediction modes. VVC uses planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes.

[0039] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of high-definition and ultra-high-definition digital video development.

[0040] Residual unit 120 may generate a residual block for a CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, residual unit 120 may generate a residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel blocks of the CU and the corresponding sample in the prediction blocks of the PUs of the CU.

[0041] The transform / quantization unit 130 may quantize the transform coefficients. The transform / quantization unit 130 may quantize the transform coefficients associated with the TUs of the CU based on a quantization parameter (QP) value associated with the CU. The video encoder 100 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0042] The inverse transform / quantization unit 140 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0043] Reconstruction unit 150 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 110 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 100 can reconstruct the pixel blocks of the CU.

[0044] The loop filter unit 160 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent coded pixels. For example, it can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with the CU.

[0045] In some embodiments, the loop filtering unit 160 includes a deblocking filtering unit and a sample adaptive offset / adaptive loop filtering (SAO / ALF) unit, wherein the deblocking filtering unit is used to remove blocking effects, and the SAO / ALF unit is used to remove ringing effects.

[0046] The decoded image buffer 170 may store the reconstructed pixel blocks. The inter-prediction unit 111 may use a reference image containing the reconstructed pixel blocks to perform inter-prediction on PUs of other images. In addition, the intra-prediction unit 112 may use the reconstructed pixel blocks in the decoded image buffer 170 to perform intra-prediction on other PUs in the same image as the CU.

[0047] The entropy encoding unit 180 may receive the quantized transform coefficients from the transform / quantization unit 130. The entropy encoding unit 180 may perform one or more entropy encoding operations on the quantized transform coefficients to generate entropy-encoded data.

[0048] FIG2 is a schematic block diagram of a video decoder according to an embodiment of the present application.

[0049] 2 , video decoder 200 includes an entropy decoding unit 210, a prediction unit 220, an inverse quantization / transformation unit 230, a reconstruction unit 240, a loop filter unit 250, and a decoded picture buffer 260. It should be noted that video decoder 200 may include more, fewer, or different functional components.

[0050] Video decoder 200 may receive a bitstream. Entropy decoding unit 210 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, entropy decoding unit 210 may parse the entropy-encoded syntax elements in the bitstream. Prediction unit 220, inverse quantization / transform unit 230, reconstruction unit 240, and loop filter unit 250 may decode video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.

[0051] In some embodiments, the prediction unit 220 includes an intra-frame prediction unit 222 and an inter-frame prediction unit 221 .

[0052] The intra-frame prediction unit 222 may perform intra-frame prediction to generate a prediction block for the PU. The intra-frame prediction unit 222 may use an intra-frame prediction mode to generate a prediction block for the PU based on the pixel blocks of spatially neighboring PUs. The intra-frame prediction unit 222 may also determine the intra-frame prediction mode of the PU based on one or more syntax elements parsed from the codestream.

[0053] The inter-frame prediction unit 221 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. In addition, if a PU is encoded using inter-frame prediction, the entropy decoding unit 210 may parse the motion information of the PU. The inter-frame prediction unit 221 may determine one or more reference blocks of the PU based on the motion information of the PU. The inter-frame prediction unit 221 may generate a prediction block for the PU based on the one or more reference blocks of the PU.

[0054] The inverse quantization / transform unit 230 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 230 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0055] After inverse quantizing the transform coefficients, the inverse quantization / transform unit 230 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0056] Reconstruction unit 240 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, reconstruction unit 240 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0057] The loop filtering unit 250 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0058] The video decoder 200 may store the reconstructed image of the CU in the decoded image buffer 260. The video decoder 200 may use the reconstructed image in the decoded image buffer 260 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0059] The basic process of video encoding and decoding is as follows: At the encoder end, an image is divided into blocks. For the current block, the prediction unit 110 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 120 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block can also be referred to as residual information. This residual block undergoes transformation and quantization by the transform / quantization unit 130, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 130 can be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 130 can be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding unit 180 receives the quantized change coefficients output by the transform and quantization unit 130, performs entropy coding on these quantized change coefficients, and outputs a bitstream. For example, the entropy coding unit 180 can eliminate character redundancy based on the target context model and probability information of the binary bitstream.

[0060] At the decoding end, the entropy decoding unit 210 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 220 uses intra-frame prediction or inter-frame prediction on the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation unit 230 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 240 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 250 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference image for inter-frame prediction of subsequent images.

[0061] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and analyzes the existing information to determine the same block division information, prediction, transform, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical.

[0062] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0063] The preceding text describes in detail the codec framework provided by the embodiments of the present application. The embodiments of the present application can be applied to the loop filtering unit in the codec framework. The following describes the loop filtering operation in detail.

[0064] In video coding standards, such as Versatile Video Coding (VVC), a loop filter unit may include filtering tools such as a deblocking filter (DBF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF).

[0065] In recent years, with the development of deep learning technology, the exploration of NNLF has gradually begun. After years of continuous research and optimization, two types of NNLF have been adopted as baseline filtering tools for neural network based video coding (NNVC) general reference software. The two NNLFs are a high complexity neural network filter and a low complexity neural network filter. The high complexity neural network filter is usually referred to as a high operation point (HOP), and Figure 3 shows a possible structure of the filter. The low complexity neural network filter is usually referred to as a low operation point (LOP), and Figure 4 shows a possible structure of the filter.

[0066] Both HOP and LOP adopt the residual network structure, taking the brightness and chrominance information of the reconstructed image and various auxiliary information as the network input, such as deblocking filter boundary strength information, quantization parameter (QP) information, etc., and there is a jump connection structure between the input and output of the network. The network output is the predicted residual image, which is added to the input reconstructed image to obtain the filtered reconstructed image.

[0067] For the predicted residual image output by the network, both HOP and LOP introduce an artificially defined image-level scaling factor (SF) to perform scaling calculations on the residual, making the final output image closer to the original image and bringing better coding performance, as shown in Figure 5.

[0068] In HOP and LOP, a calculation formula is defined based on the least squares method to derive the scaling factor. The following details the derivation process of the scaling factor provided by related technologies.

[0069] Assume that the width of an image is W and the height is H, and P(x,y) represents the coordinates of the pixel point P. The reconstructed image input to the neural network is rec(x,y), the residual image output by the neural network is res(x,y), the filtered image finally output by the neural network is cnn(x,y), and the original image is org(x,y).

[0070] For the image-level scaling factor, the residual between the corresponding pixels of the two images is first calculated based on the following equations 1 and 2. i ,y i )=[org(x i ,y i )-rec(x i ,y i )] (Formula 1) cnnResi(x i ,y i )=[cnn(x i ,y i )-rec(x i ,y i )] (Formula 2)

[0071] Then, based on Equations 3 to 6, all pixels of the current image are traversed, and residual-related information of the entire image is obtained through weighted sums in various forms.

[0072] Finally, the least squares method is used to calculate the statistical one-dimensional scaling factor for the image level, as shown in Equation 7, where SF bottom and SF up These are artificially set lower and upper limits.

[0073] According to the derived scaling factor, the residual of the reconstructed image and the filtered image is corrected to obtain the final output image output(x i ,y i ), as shown in Equation 8. As for the scaling factor itself, after being derived at the encoding end, it needs to be encoded with a certain number of bits and then parsed at the decoding end before it can be used. output(x i ,y i )=rec(x i ,yi )+SF*[cnn(x i ,y i )-rec(x i ,y i )] (Equation 8)

[0074] In the current calculation process of the scaling factor, it is assumed that the prediction residual output by the NNLF and the residual between the original image and the reconstructed image conform to the law of linear distribution, and then the least squares method is used to fit and solve, as shown in Equation 9.

[0075] However, as shown in Figure 6, through simulation experiments and data statistical analysis of the distribution of residuals, it is found that the residuals between the original image and the reconstructed image, as well as the predicted residuals output by the NNLF, do not conform to a simple linear law and are difficult to fit using a single function curve.

[0076] In statistics, the Pearson correlation coefficient (PCC) can be used to measure the correlation (linear correlation) between two samples X and Y. It is calculated as the quotient of the covariance and standard deviation between the two samples, as shown in Formula 10, where and represents the mean of the two samples.

[0077] According to the definition of the Pearson correlation coefficient, when |r| > 0.5, it indicates that the two samples have a certain linear relationship, making linear regression worthwhile. However, when calculating the residual between the original image and the reconstructed image, and then calculating the Pearson correlation coefficient between this residual and the predicted residual output by the NNLF, the Pearson correlation coefficient is found to be around 0.25, indicating that the relationship between the two is not a simple linear relationship. Therefore, using the least squares method to derive the scaling factor is inaccurate.

[0078] Based on this, an embodiment of the present application proposes an encoding method, including: determining a reconstructed image of an original image; performing loop filtering on the reconstructed image according to a neural network to determine a first residual image; correcting the first residual image according to multiple candidate values ​​of a first scaling factor to determine multiple output images; and determining a target value of the first scaling factor from the multiple candidate values ​​based on the multiple output images.

[0079] In addition, an embodiment of the present application also proposes a decoding method, including: parsing a code stream to determine first information; determining a target value of the first scaling factor from multiple candidate values ​​of the first scaling factor based on the first information; performing loop filtering on the reconstructed image based on a neural network to determine a first residual image; and correcting the first residual image according to the target value of the first scaling factor to determine a reconstructed image after loop filtering.

[0080] As can be seen from the above content, the embodiment of the present application determines the target value of the scaling factor from multiple candidate values ​​based on the correction results of the multiple candidate values ​​of the scaling factor. This makes the value of the scaling factor more accurate and helps improve the quality of loop filtering.

[0081] The following first describes in detail the decoding method provided in the embodiment of the present application.

[0082] FIG7 is a flow chart of a decoding method according to an embodiment of the present application. The method of FIG7 can also be referred to as a loop filtering method. The method of FIG7 can be applied to a decoder, for example, to a loop filtering unit of a decoder.

[0083] 7 , in steps S710 to S720 , the code stream is parsed to determine first information; and based on the first information, a target value of the first scaling factor is determined from a plurality of candidate values ​​of the first scaling factor.

[0084] In some implementations, the first scaling factor may be referred to as a refined scaling factor (RSF).

[0085] In some implementations, the first scaling factor may correspond to a frame of image. Thus, the first scaling factor may be the scaling factor corresponding to the current frame. The first scaling factor may be referred to as an image-level scaling factor.

[0086] In some implementations, the first scaling factor may correspond to a color component (or color channel), and different color components may correspond to different first scaling factors. For example, if the color information of a pixel is represented by three color components, Y, Cb, and Cr, then Y, Cb, and Cr may each have a corresponding first scaling factor. For another example, if the color information of a pixel is represented by two color channels, luma and chroma, then each luma and chroma channel may each have a corresponding first scaling factor.

[0087] In some implementations, the first scaling factor may correspond to an image block in a frame of image. Therefore, the first scaling factor may be the scaling factor corresponding to the current block.

[0088] In some implementations, the target value of the first scaling factor may be determined based on (eg, selected from) multiple candidate values ​​of the first scaling factor.

[0089] In some implementations, the multiple candidate values ​​of the first scaling factor are predefined values. For example, the number and values ​​of the candidate values ​​of the first scaling factor may be agreed upon by a standard.

[0090] In some implementations, the multiple candidate values ​​for the first scaling factor may be determined based on certain rules. For example, the multiple candidate values ​​for the first scaling factor may be determined by traversing the range of values ​​of the first scaling factor at a traversal step size (the above traversal process may be implemented at the encoder, and the decoder may directly determine the value of the first scaling factor based on information in the bitstream).

[0091] In some implementations, the above traversal may use a certain boundary value (such as an upper limit or a lower limit) of the first scaling factor as an initial value.

[0092] For example, it is assumed that the value range of RSF (ie, the first scaling factor) is (SF bottom ,SF up ), the traversal step length is d, then the number of traversals required is (if If it is not an integer, then Round down), the RSF obtained by the i-th traversal i The candidate value is SF bottom +i*d. RSF i Substituting into the formula 8 mentioned above, we can get RSF i The corresponding output image. Then, RSF i The corresponding output image is compared with the original image and the RSF is calculated i The corresponding distortion cost D i .

[0093] Calculate the minimum distortion cost D of RSF through K traversals i Then, the minimum distortion cost D can be i The corresponding RSF value is used as the target value of RSF.

[0094] The above export method can be implemented using the following pseudo code:

[0095] The meaning of each parameter in the above pseudo code can be found in the relevant introduction of Formulas 1 to 8 above, and MSE(.) represents the minimum square error.

[0096] The traversal step mentioned above may be a fixed step. Alternatively, in some implementations, the traversal step mentioned above may also be a non-fixed step (i.e., a traversal step of non-equal length). For example, the value range of the first scaling factor may include a first value interval and a second value interval, the first value interval corresponding to a first traversal step, the second value interval corresponding to a second traversal step, and the first traversal step and the second traversal step are different. The use of different traversal step sizes helps improve encoding performance. For details, please refer to the description of the encoding side later.

[0097] In some implementations, the first information (which may be represented by picture_rsf_value) is used to determine a target value of the first scaling factor.

[0098] In some implementations, the first information is a target value of the first scaling factor. That is, the value of the first scaling factor can be directly carried in the bitstream.

[0099] In some implementations, the first information may be index information of a target value of the first scaling factor among multiple candidate values ​​of the first scaling factor. For example, if the first scaling factor includes 10 candidate values ​​and the first information is the third candidate value among the 10 candidate values, the value of the first information may be 3. Compared to directly carrying the value of the first scaling factor in the bitstream, carrying the index information can reduce the number of coding bits required to carry the first scaling factor in the bitstream.

[0100] In some implementations, the first information is used to indicate or determine the number of traversals corresponding to the first scaling factor (the traversal number can be understood as an implementation of index information). For example, if the encoder uses the traversal method described above to determine candidate values ​​for the first scaling factor, the first information can indicate the number of traversals corresponding to the first scaling factor. After determining the number of traversals corresponding to the first scaling factor, a target value for the first scaling factor can be determined based on the initial value of the first scaling factor (the initial value can be a boundary value of the first scaling factor, such as an upper limit or lower limit), the traversal step size, and the number of traversals.

[0101] For example, suppose the RSF obtained in the i-th traversal is i The candidate value of SF bottom +i*d, and the target value of RSF is the candidate value RSF obtained by the kth traversal k , then the first information can indicate the value of k. In this way, at the decoding end, it can be based on SF bottom +k*d determines the target value of RSF.

[0102] In the related art, for the calculated value of the scaling factor, N (N=9) bits are required to directly encode the value of the scaling factor. Unlike the related art, the embodiment of the present application introduces the concept of traversal step size, so that the quantization scale can be amplified, and only the number of traversals needs to be represented by encoding. Compared with the method of directly encoding the value of the scaling factor, encoding the number of traversals can reduce the number of coding bits required for the scaling factor. The calculation method of the number of coding bits for the first scaling factor is shown in Equation 11:

[0103] Here, M represents the number of coding bits of the first scaling factor, and k represents the number of traversals corresponding to the first scaling factor (ie, the target value of the first scaling factor is obtained after k traversals).

[0104] In step S730, loop filtering is performed on the reconstructed image according to the neural network to determine a first residual image.

[0105] In some implementations, the neural network can be a HOP.

[0106] In some implementations, the neural network can be a LOP.

[0107] In some implementations, the reconstructed image may be a frame of image.

[0108] In some implementations, the reconstructed image may be an image block in a frame of image.

[0109] In some implementations, the first residual image can be referred to as a predicted residual image of the neural network.

[0110] In some implementations, step S730 may include: performing loop filtering on the reconstructed image according to the neural network to determine a filtered image; and determining a first residual image according to the filtered image and the reconstructed image. The filtered image mentioned here refers to the image output by the neural network.

[0111] In some implementations, determining the first residual image based on the filtered image and the reconstructed image may include: determining the difference between the filtered image and the reconstructed image (ie, calculating the difference between corresponding pixels) as the first residual image.

[0112] In step S740 , the first residual image is corrected according to the target value of the first scaling factor to determine a reconstructed image after loop filtering.

[0113] In some implementations, the first residual image may be corrected according to the target value of the first scaling factor to determine a corrected residual image; and then, a reconstructed image after loop filtering may be determined based on the corrected residual image and the reconstructed image.

[0114] In some implementations, correcting the first residual image according to a target value of a first scaling factor to determine a corrected residual image may include scaling each pixel in the first residual image according to the target value of the first scaling factor (i.e., multiplying each pixel in the first residual image by the first scaling factor), thereby determining the corrected residual image.

[0115] In some implementations, determining the reconstructed image after loop filtering based on the corrected residual image and the reconstructed image may include: summing the corrected residual image and the reconstructed image (corresponding to pixel summation), and using the summation result as the reconstructed image after loop filtering.

[0116] In some implementations, the code stream may be parsed to determine the second information. The second information may be used to indicate whether to correct the first residual image based on the first scaling factor. Alternatively, the second information may be used to indicate whether to perform loop filtering using the optimized scaling factor provided in embodiments of the present application.

[0117] In some implementations, the second information may be represented by, for example, picture_rsf_enable_flag. The second information may include a first value and a second value. The first value may be, for example, true or 1. The first value may indicate that the first residual image is corrected based on the first scaling factor. The second value may be, for example, false or 0. The second value may indicate that the first residual image is not corrected based on the first scaling factor.

[0118] In some implementations, if the first residual image is not corrected based on the first scaling factor, the first residual image can be corrected based on a scaling factor provided by related art. In this embodiment of the present application, the scaling factor provided by related art is referred to as a second scaling factor. The value of the second scaling factor is determined differently from the value of the first scaling factor. The value of the second scaling factor can be determined using a least squares method, as described in detail in Equations 1 to 8 above. Accordingly, if the first residual image is corrected using the second scaling factor, the bitstream can be parsed to determine third information. The third information can be a target value for the second scaling factor.

[0119] The following describes the embodiments of the present application in more detail with reference to specific examples. The following examples are applicable to the loop filter unit of the decoding end. It should be noted that the following examples are merely intended to help those skilled in the art understand the embodiments of the present application and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Based on the examples provided, those skilled in the art can obviously make various equivalent modifications or variations, and such modifications or variations also fall within the scope of the embodiments of the present application.

[0120] The decoder acquires and parses the bitstream. When parsed to the loop filter unit, it can be processed according to the filter sequence specified by the loop filter unit (such as DBF, SAO, etc.). When entering the scaling factor optimization unit, it can be executed according to the process.

[0121] Step a): For the current color component of the current frame in the current sequence, the image-level scaling factor optimization enable flag, picture_rsf_enable_flag, may be parsed. If picture_rsf_enable_flag is "1," the RSF (corresponding to the first scaling factor mentioned above) value, picture_rsf_value, may be further parsed. If picture_rsf_enable_flag is "0," the target SF (corresponding to the second scaling factor mentioned above) value, picture_sf_value, may be further parsed. After completing step a), proceed to step b).

[0122] Step b): For the current color component of the current frame of the current sequence, the residual between the reconstructed image and the filtered image (corresponding to the first residual image mentioned above) is corrected using Equation 8 to obtain an output image. After executing step b), jump to step c).

[0123] Step c): If the current frame has completed processing by the scaling factor optimization unit, the next frame image is loaded and the process jumps to step a) to continue processing.

[0124] The image header can be defined as follows:

[0125] In the syntax of the above image header, picture_rsf_enable_flag[N] indicates the flag that enables picture-level scaling factor optimization. When picture_rsf_enable_flag is 1, it indicates that RSF is allowed; when picture_rsf_enable_flag is 0, it indicates that RSF is not allowed, and SF can be used. In addition, the image header further defines picture_rsf_value[N] and picture_sf_value[N]. picture_rsf_value[N] indicates the value of RSF; picture_sf_value[N] indicates the value of SF. The value of N can be 3, indicating the three color components Y, Cb, and Cr. Alternatively, the value of N can be 2, indicating the two color channels of luminance and chrominance.

[0126] The decoding method provided by the embodiment of the present application is described in detail above in conjunction with Figure 7. The encoding method provided by the embodiment of the present application is described in detail below in conjunction with Figure 8.

[0127] FIG8 is a flow chart of an encoding method according to an embodiment of the present application. The method of FIG8 can also be referred to as a loop filtering method. The method of FIG8 can be applied to an encoder, for example, to a loop filtering unit of an encoder.

[0128] 8 , in step S810 , a reconstructed image of the original image is determined.

[0129] In some implementations, the original image or the reconstructed image may be a frame of image.

[0130] In some implementations, the original image or the reconstructed image may be an image block in a frame of image.

[0131] In some implementations, step S810 may include: performing prediction, transformation, quantization, and inverse quantization operations on the original image to determine a predicted image and a residual image after inverse quantization; and then determining a reconstructed image based on the predicted image and the residual image. The reconstructed image may be, for example, the sum of the predicted image and the residual image.

[0132] In step S820, loop filtering is performed on the reconstructed image according to the neural network to determine a first residual image.

[0133] In some implementations, the neural network can be a HOP.

[0134] In some implementations, the neural network can be a LOP.

[0135] In some implementations, step S820 may include: performing loop filtering on the reconstructed image according to the neural network to determine a filtered image; and determining a first residual image based on the filtered image and the reconstructed image. The filtered image referred to herein refers to the image output by the neural network. The first residual image may also be referred to as a predicted residual image of the neural network.

[0136] In some implementations, determining the first residual image based on the filtered image and the reconstructed image may include: determining the difference between the filtered image and the reconstructed image (ie, calculating the difference between corresponding pixels) as the first residual image.

[0137] In step S830, the first residual image is corrected according to multiple candidate values ​​of the first scaling factor to determine multiple output images.

[0138] In step S840 , a target value of the first scaling factor is determined from a plurality of candidate values ​​according to the plurality of output images.

[0139] In some implementations, the first scaling factor may be referred to as RSF.

[0140] In some implementations, the first scaling factor may correspond to a frame of image. Therefore, the first scaling factor may be a scaling factor corresponding to the current frame. In other words, the first scaling factor may be an image-level scaling factor.

[0141] In some implementations, the first scaling factor may correspond to an image block in a frame of image.

[0142] In some implementations, the first scaling factor may correspond to a color component (or color channel), and different color components may correspond to different first scaling factors. For example, if the color information of a pixel is represented by three color components (Y, Cb, and Cr), each of these components may have a corresponding first scaling factor. For another example, if the color information of a pixel is represented by two color channels (luminance and chrominance), each of these channels may have a corresponding first scaling factor.

[0143] In some implementations, the multiple candidate values ​​of the first scaling factor are predefined values. For example, the number and values ​​of the candidate values ​​of the first scaling factor may be agreed upon by a standard.

[0144] In some implementations, the multiple candidate values ​​of the first scaling factor may be determined based on certain rules. For example, the multiple candidate values ​​of the first scaling factor may be determined by traversing the range of values ​​of the first scaling factor according to a traversal step size.

[0145] In some implementations, step S840 may include: determining costs corresponding to multiple candidate values ​​based on the multiple output images and the original image; and determining a target value from the multiple candidate values ​​based on the costs corresponding to the multiple candidate values. The cost mentioned here may refer to a distortion cost, for example. After determining the costs corresponding to the multiple candidate values, the candidate value with the smallest corresponding cost may be determined as the target value of the first scaling factor.

[0146] In some implementations, the above traversal may use a certain boundary value (such as an upper limit or a lower limit) of the first scaling factor as an initial value.

[0147] For example, it is assumed that the value range of RSF (ie, the first scaling factor) is (SF bottom ,SF up ), the traversal step length is d, then the number of traversals required is (if If it is not an integer, then Round down), the RSF obtained by the i-th traversal i The candidate value is SF bottom +i*d. RSF i Substituting into Equation 8, we can get RSF iThe corresponding output image. Then, RSF i The corresponding output image is compared with the original image and the RSF is calculated i The corresponding distortion cost D i .

[0148] Calculate the minimum distortion cost D of RSF through K traversals i Then, the minimum distortion cost D can be i The corresponding RSF value is the target value of RSF.

[0149] The above export method can be implemented using the following pseudo code:

[0150] The meaning of each parameter in the above pseudo code can be found in the relevant introduction of Formulas 1 to 8 above, and MSE(.) represents the minimum square error.

[0151] In some implementations, the traversal step mentioned above may be a fixed step.

[0152] In practice, statistical analysis reveals that the values ​​of the first scaling factor are primarily concentrated within certain specific ranges. Therefore, in some implementations, the traversal step size can be set to a non-fixed step size (i.e., a traversal step size of varying lengths) to increase flexibility. For example, the value range of the first scaling factor includes a first value interval and a second value interval, where the first value interval corresponds to a first traversal step size, and the second value interval corresponds to a second traversal step size, where the first traversal step size and the second traversal step size are different.

[0153] For example, the first value interval can be a low-frequency range where the first scaling factor may appear, and the second value interval can be a high-frequency range where the first scaling factor may appear. For the low-frequency range, a larger traversal step size d1 can be set to perform a coarser-grained search; for the high-frequency range, a smaller traversal step size d2 can be set to perform a finer-grained search.

[0154] As a more specific example, suppose that through statistical analysis, it is determined that the high-frequency range in which the scaling factor may appear is SF r1 ~SF r2 , then for SF bottom ~SF r1 interval, d1 (d1 is greater than the fixed step size d mentioned above) can be used as the traversal step size. At this time, the RSF i =SF bottom +i*d1; for SF r1 ~SF r2 interval, d2 (d2 is greater than the fixed step size d mentioned above) can be used as the traversal step size. At this time, the RSFi =SF r1 +i*d2; for SF r2 ~SF up interval, d1 can be used as the traversal step, and the RSF i =SF r2 +i*d1. By controlling d1 and d2, as well as the high frequency range SF r1 ~SF r2 The interval can keep the number of traversals basically consistent with that when using a fixed step size d, that is, the number of bits required for encoding remains unchanged, and the RSF search is more finely adjusted to further improve the encoding performance.

[0155] In some implementations, the first information may be written into a code stream.

[0156] In some implementations, the first information may be represented by picture_rsf_value.

[0157] In some implementations, the first information is a target value of the first scaling factor. That is, the value of the first scaling factor can be directly carried in the bitstream.

[0158] In some implementations, the first information may be index information of a target value of the first scaling factor among multiple candidate values ​​of the first scaling factor. For example, if the first scaling factor includes 10 candidate values ​​and the first information is the third value among the 10 candidate values, the value of the first information may be 3. Compared to directly carrying the value of the first scaling factor in the bitstream, carrying the index information can reduce the number of coding bits required to carry the first scaling factor in the bitstream.

[0159] In some implementations, the first information is used to indicate the number of traversals corresponding to the first scaling factor (the number of traversals can be understood as an implementation of an index value). For example, if the encoder uses the traversal method mentioned above to determine the candidate value of the first scaling factor, the first information can indicate the number of traversals corresponding to the first scaling factor. For example, assuming that the RSF obtained in the i-th traversal is i The candidate value of SF bottom +i*d, and the target value of RSF is the candidate value RSF obtained by the kth traversal k , then the first information can indicate the value of k. In this way, at the decoding end, it can be based on SF bottom +k*d determines the target value of RSF.

[0160] In related art, the calculated scaling factor value requires N (N=9) bits to directly encode the scaling factor value. Unlike related art, the present embodiment introduces the concept of traversal step size, thereby amplifying the quantization scale and only requiring encoding to represent the number of traversals. Compared to directly encoding the scaling factor value, encoding the number of traversals can reduce the number of bits required to encode the scaling factor. The calculation method for the number of encoding bits for the first scaling factor can be found in Equation 11 above.

[0161] In some implementations, step S830 may include: correcting the first residual image according to multiple candidate values ​​of the first scaling factor to determine multiple corrected residual images; and determining multiple output images according to the multiple corrected residual images and the reconstructed image.

[0162] For example, each pixel in the first residual image may be scaled according to multiple candidate values ​​of the first scaling factor (i.e., each pixel in the first residual image is multiplied by the first scaling factor), thereby determining multiple modified residual images (corresponding one-to-one with the multiple candidate values). Subsequently, the multiple modified residual images may be summed with the reconstructed image to determine multiple output images (corresponding one-to-one with the multiple candidate values).

[0163] In some implementations, a second residual image can be determined based on the original image and the reconstructed image. Then, a target value for a second scaling factor can be determined based on the first residual image and the second residual image. The second scaling factor mentioned herein may refer to a scaling factor provided by related art, i.e., a scaling factor derived by least squares method. The method for determining the second scaling factor can be found in Equations 1 to 8 above. After determining the target value for the second scaling factor, a target scaling factor can be determined from the first scaling factor and the second scaling factor based on the target value for the first scaling factor and the target value for the second scaling factor.

[0164] For example, the first residual image can be corrected based on the target value of the second scaling factor to determine the first output image. Then, a target scaling factor can be determined from the first scaling factor and the second scaling factor based on the first and second output images, with the second output image being the output image corresponding to the target value of the first scaling factor. Exemplarily, the cost (e.g., distortion cost) corresponding to the first scaling factor can be determined based on the first output image and the original image; the cost (e.g., distortion cost) corresponding to the second scaling factor can be determined based on the second output image and the original image; and the target scaling factor can be determined from the first and second scaling factors based on the cost corresponding to the first scaling factor and the cost corresponding to the second scaling factor (e.g., the scaling factor with the smallest corresponding cost is used as the target scaling factor).

[0165] In some implementations, the second information is written into the bitstream. The second information is used to indicate whether to correct the first residual image based on the first scaling factor. In other words, the second information can be used to indicate whether to perform loop filtering using the optimized scaling factor provided in the embodiments of the present application.

[0166] In some implementations, the second information may be represented by, for example, picture_rsf_enable_flag.

[0167] The second information may include a first value and a second value. The first value may be, for example, true or 1. The first value may indicate that the first residual image is corrected based on the first scaling factor. The second value may be, for example, false or 0. The second value may indicate that the first residual image is not corrected based on the first scaling factor.

[0168] In some implementations, if the first residual image is not corrected based on the first scaling factor, the first residual image may be corrected based on a scaling factor provided by related technologies. Accordingly, if the first residual image is corrected using a second scaling factor, third information may be written into the bitstream, where the third information is a target value for the second scaling factor.

[0169] The following describes the embodiments of the present application in more detail with reference to specific examples. The following examples are applicable to the loop filtering unit of the encoding end. It should be noted that the following examples are merely intended to help those skilled in the art understand the embodiments of the present application and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Based on the examples provided, those skilled in the art can obviously make various equivalent modifications or variations, and such modifications or variations also fall within the scope of the embodiments of the present application.

[0170] This solution proposes an optimization method for image-level scaling factor, which optimizes the image-level scaling factor through designed traversal and encoding algorithms to obtain better encoding performance.

[0171] As shown in Figure 9, the scaling factor optimization unit (i.e., the unit that calculates the RSF) is located in the codec's loop filter unit 160 (see Figure 1 for an introduction to the other units in Figure 9). The use of scaling factors is independent of the use of DBF, SAO, and ALF. The scaling factor optimization unit is positioned after the NNLF and before the ALF.

[0172] The scaling factor optimization unit modifies the scaling factor derivation method used by NNLF. On the encoder side, the rate-distortion cost of the RSF and SF used in this paper are compared to determine which scaling factor to use. The selected scaling factor is encoded into the bitstream for the decoder to read. On the decoder side, once the scaling factor is parsed, the output image is processed.

[0173] When the encoding end enters the loop filter unit, it is processed according to the specified filter order. When it enters the scaling factor optimization unit, it is processed according to the following process.

[0174] Step a): For the current color component of the current frame of the current sequence, traverse all pixels in the corresponding image and determine SF using equations 1 to 7. After step a), jump to step b).

[0175] Step b): For the current color component of the current frame of the current sequence, traverse all pixels in the corresponding image to determine the RSF. After step b) is completed, jump to step c).

[0176] Step c): According to SF and RSF, the residuals of the reconstructed image and the filtered image (corresponding to the first residual image in the previous text) are corrected by formula (8) to obtain two output images. The two output images are compared with the original image and the cost, such as the distortion cost D, is calculated. The two costs are compared. If D RSF <D SF , then RSF is used as the final scaling factor for the current color component of the current frame; otherwise, if D RSF >=D SF , then SF is used as the final scaling factor. After step c) is completed, jump to step d).

[0177] Step d): Encode the picture-level scaling factor optimization enable flag (picture_rsf_enable_flag) for the current frame into the bitstream. If picture_rsf_enable_flag is "1," further encode the target RSF value (picture_rsf_value); if picture_rsf_enable_flag is "0," further encode the target SF value (picture_sf_value). After step d), proceed to step e).

[0178] Step e): If the current frame has completed processing by the scaling factor optimization unit, the next frame is loaded for processing and the process jumps to step a).

[0179] The following describes the test results of the above example.

[0180] Based on HOP NNLF, this paper implements the scheme based on all frame types (including I-frames, P-frames, and B-frames) and tests the performance. In the specific simulation experiment, the traversal step size is set to 5, so the number of bits required to encode the RSF is M (M=6).

[0181] Under the general test conditions of Random Access, Low Delay B, All Intra configuration, the common sequences specified by the Joint Video Experts Team (JVET) were tested, and the SFs provided by related technologies were used for comparison. The performance of some sequences was obtained, and the results are shown in Tables 1, 2, and 3.

[0182] Table 1 Random Access Performance

[0183] Table 2 All Intra performance

[0184] Table 3 Low Delay B performance

[0185] Analyzing the data in the above table, we can see that by introducing RSF, the coding performance can be further improved on the basis of NNLF, especially in the chrominance component of LDB, and at the same time, this scheme does not increase the decoding complexity.

[0186] The method embodiment of the present application is described in detail above in conjunction with Figures 1 to 9 . The device embodiment of the present application is described in detail below in conjunction with Figures 10 to 13 . It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for portions not described in detail, reference can be made to the above method embodiment.

[0187] FIG10 is a schematic diagram of the structure of a decoder provided by an embodiment of the present application. The decoder 1000 of FIG10 includes a parsing unit 1010 and a filtering unit 1020.

[0188] The parsing unit 1010 is configured to parse the code stream and determine the first information.

[0189] The filtering unit 1020 is configured to determine a target value of the first scaling factor from multiple candidate values ​​of the first scaling factor based on the first information; perform loop filtering on the reconstructed image according to the neural network to determine a first residual image; and correct the first residual image according to the target value of the first scaling factor to determine a reconstructed image after loop filtering.

[0190] In some implementations, the filtering unit 1020 is configured to: determine a number of traversals corresponding to the first scaling factor based on the first information; and determine a target value of the first scaling factor based on an initial value of the first scaling factor, a traversal step size, and the number of traversals.

[0191] In some implementations, the value range of the first scaling factor includes a first value interval and a second value interval, the first value interval corresponds to a first traversal step, the second value interval corresponds to a second traversal step, and the first traversal step and the second traversal step are different.

[0192] In some implementations, the first information is used to indicate the number of traversals corresponding to the first scaling factor.

[0193] In some implementations, the first information is index information of the target value among the multiple candidate values.

[0194] In some implementations, the parsing unit 1010 is further configured to: parse the code stream to determine second information, where the second information is used to indicate whether to correct the first residual image based on the first scaling factor.

[0195] In some implementations, the second information includes a first value and a second value, the first value indicating that the first residual image is corrected based on the first scaling factor, and the second value indicating that the first residual image is not corrected based on the first scaling factor.

[0196] In some implementations, the parsing unit 1010 is further configured to: if the value of the second information is the second value, determine third information, where the third information is a target value of a second scaling factor, and the second scaling factor is determined differently from the first scaling factor.

[0197] In some implementations, the filtering unit 1020 is configured to: perform loop filtering on the reconstructed image according to the neural network to determine a filtered image; and determine the first residual image according to the filtered image and the reconstructed image.

[0198] In some implementations, the filtering unit 1020 is configured to: correct the first residual image according to the target value of the first scaling factor to determine a corrected residual image; and determine the reconstructed image after loop filtering based on the corrected residual image and the reconstructed image.

[0199] In some implementations, the reconstructed image is a frame of image or an image block in a frame of image.

[0200] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0201] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0202] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the decoder 1000. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the decoding method in the first embodiment.

[0203] Based on the composition of the above-mentioned decoder 1000 and the computer-readable storage medium, refer to Figure 11, which shows a specific hardware structure diagram of the decoder 1000 provided in an embodiment of the present application. As shown in Figure 11, the decoder 1100 may include: a communication interface 1110, a memory 1120 and a processor 1130; each component is coupled together through a bus system 1140. It can be understood that the bus system 1140 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1140 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 1140 in Figure 11. Among them,

[0204] Communication interface 1110, used for sending and receiving signals when sending and receiving information with other external network elements;

[0205] Memory 1120, for storing computer programs;

[0206] The processor 1130 is configured to, when running the computer program, execute:

[0207] Parsing the code stream to determine the first information;

[0208] determining, according to the first information, a target value of the first scaling factor from a plurality of candidate values ​​of the first scaling factor;

[0209] Performing loop filtering on the reconstructed image according to the neural network to determine a first residual image;

[0210] The first residual image is corrected according to the target value of the first scaling factor to determine a reconstructed image after loop filtering.

[0211] It is understood that the memory 1120 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1120 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0212] The processor 1130 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1130. The above-mentioned processor 1130 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1120 , and the processor 1130 reads the information in the memory 1120 and completes the steps of the above method in combination with its hardware.

[0213] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0214] Optionally, as another embodiment, the processor 1130 is further configured to execute the decoding method described in the above embodiment when running the computer program.

[0215] FIG12 is a schematic diagram of the structure of an encoder provided by an embodiment of the present application. As shown in FIG12 , the encoder 1200 includes: a determination unit 1210 and a filtering unit 1220 .

[0216] The determining unit 1210 is configured to determine a reconstructed image of the original image.

[0217] The filtering unit 1220 is configured to perform loop filtering on the reconstructed image according to a neural network to determine a first residual image; correct the first residual image according to multiple candidate values ​​of the first scaling factor to determine multiple output images; and determine a target value of the first scaling factor from the multiple candidate values ​​based on the multiple output images.

[0218] In some implementations, the multiple candidate values ​​are determined based on the following method: traversing within a value range of the first scaling factor according to a traversal step size.

[0219] In some implementations, the value range of the first scaling factor includes a first value interval and a second value interval, the first value interval corresponds to a first traversal step, the second value interval corresponds to a second traversal step, and the first traversal step and the second traversal step are different.

[0220] In some implementations, the encoder 1200 further includes: a first encoding unit configured to write first information into a bitstream, where the first information is used to indicate a number of traversals corresponding to the first scaling factor.

[0221] In some implementations, the filtering unit 1220 is further configured to determine a second residual image based on the original image and the reconstructed image; determine a target value of a second scaling factor based on the first residual image and the second residual image; and determine a target scaling factor from the first scaling factor and the second scaling factor based on the target value of the first scaling factor and the target value of the second scaling factor.

[0222] In some implementations, the filtering unit 1220 is configured to: correct the first residual image according to the target value of the second scaling factor to determine a first output image; and determine the target scaling factor from the first scaling factor and the second scaling factor based on the first output image and the second output image, where the second output image is the output image corresponding to the target value of the first scaling factor.

[0223] In some implementations, the filtering unit 1220 is configured to: determine a cost corresponding to the first scaling factor based on the first output image and the original image; determine a cost corresponding to the second scaling factor based on the second output image and the original image; and determine the target scaling factor from the first scaling factor and the second scaling factor based on the cost corresponding to the first scaling factor and the cost corresponding to the second scaling factor.

[0224] In some implementations, the encoder 1200 further includes: a second encoding unit configured to write first information into a bitstream, where the first information is index information of the target value among the multiple candidate values.

[0225] In some implementations, the encoder 1200 further includes: a third encoding unit configured to write second information into a bitstream, where the second information is used to indicate whether to correct the first residual image based on the first scaling factor.

[0226] In some implementations, the second information includes a first value and a second value, the first value indicating that the first residual image is corrected based on the first scaling factor, and the second value indicating that the first residual image is not corrected based on the first scaling factor.

[0227] In some implementations, if the value of the second information is the second value, third information is written into the codestream, where the third information is a target value of a second scaling factor, the second scaling factor is determined based on the first residual image and the second residual image, and the second residual image is determined based on the original image and the reconstructed image.

[0228] In some implementations, the filtering unit 1220 is configured to: perform loop filtering on the reconstructed image according to the neural network to determine a filtered image; and determine the first residual image according to the filtered image and the reconstructed image.

[0229] In some implementations, the filtering unit 1220 is configured to: correct the first residual image according to multiple candidate values ​​of the first scaling factor to determine multiple corrected residual images; and determine the multiple output images based on the multiple corrected residual images and the reconstructed image.

[0230] In some implementations, the filtering unit 1220 is configured to: determine the costs corresponding to the multiple candidate values ​​based on the multiple output images and the original image; and determine the target value from the multiple candidate values ​​based on the costs corresponding to the multiple candidate values.

[0231] In some implementations, the reconstructed image is a frame of image or an image block in a frame of image.

[0232] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0233] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0234] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 1200. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the decoding method described in any one of the aforementioned embodiments.

[0235] Based on the composition of the above-mentioned encoder 1200 and the computer-readable storage medium, refer to Figure 13, which shows a specific hardware structure diagram of the encoder 1200 provided in an embodiment of the present application. As shown in Figure 13, the encoder 1200 may include: a communication interface 1310, a memory 1320 and a processor 1330; each component is coupled together through a bus system 1340. It can be understood that the bus system 1340 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1340 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 1340 in Figure 13. Among them,

[0236] The communication interface 1310 is used to receive and send signals when sending and receiving information with other external network elements;

[0237] Memory 1320, for storing computer programs;

[0238] The processor 1330 is configured to, when running the computer program, execute:

[0239] determining a reconstructed image of the original image;

[0240] performing loop filtering on the reconstructed image according to a neural network to determine a first residual image;

[0241] Correcting the first residual image according to multiple candidate values ​​of the first scaling factor to determine multiple output images;

[0242] A target value of the first scaling factor is determined from the plurality of candidate values ​​according to the plurality of output images.

[0243] It is understood that the memory 1320 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1320 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0244] The processor 1330 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1330. The above-mentioned processor 1330 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1320 , and the processor 1330 reads the information in the memory 1320 and completes the steps of the above method in combination with its hardware.

[0245] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0246] Optionally, as another embodiment, the processor 1330 is further configured to execute the encoding method in the aforementioned embodiment when running the computer program.

[0247] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0248] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0249] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0250] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0251] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0252] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A decoding method, applied to a decoder, comprising: Parsing the code stream to determine the first information; determining, according to the first information, a target value of the first scaling factor from a plurality of candidate values ​​of the first scaling factor; Performing loop filtering on the reconstructed image according to the neural network to determine a first residual image; The first residual image is corrected according to the target value of the first scaling factor to determine a reconstructed image after loop filtering.

2. The method according to claim 1, wherein: The determining, according to the first information, a target value of the first scaling factor from a plurality of candidate values ​​of the first scaling factor includes: Determine, according to the first information, a number of traversals corresponding to the first scaling factor; A target value of the first scaling factor is determined according to an initial value of the first scaling factor, a traversal step, and the number of traversals.

3. The method according to claim 2, wherein: The value range of the first scaling factor includes a first value interval and a second value interval, the first value interval corresponds to a first traversal step length, the second value interval corresponds to a second traversal step length, and the first traversal step length is different from the second traversal step length.

4. The method according to claim 1, wherein: The first information is index information of the target value among the multiple candidate values.

5. The method according to claim 1, wherein: The method further comprises: Parse the code stream to determine second information, where the second information is used to indicate whether to correct the first residual image based on the first scaling factor.

6. The method according to claim 5, wherein: The second information includes a first value and a second value, the first value indicates that the first residual image is corrected based on the first scaling factor, and the second value indicates that the first residual image is not corrected based on the first scaling factor.

7. The method according to claim 6, wherein: If the value of the second information is the second value, the method further includes: The code stream is parsed to determine third information, where the third information is a target value of the second scaling factor.

8. The method according to claim 1, wherein: The step of performing loop filtering on the reconstructed image according to the neural network to determine the first residual image includes: Performing loop filtering on the reconstructed image according to the neural network to determine a filtered image; The first residual image is determined according to the filtered image and the reconstructed image.

9. The method according to claim 1, wherein: The step of correcting the first residual image according to the target value of the first scaling factor to determine a reconstructed image after loop filtering includes: Correcting the first residual image according to the target value of the first scaling factor to determine a corrected residual image; The reconstructed image after loop filtering is determined according to the corrected residual image and the reconstructed image.

10. The method according to claim 1, wherein: The reconstructed image is a frame of image or an image block in a frame of image.

11. A coding method, applied to an encoder, comprising: determining a reconstructed image of the original image; Performing loop filtering on the reconstructed image according to a neural network to determine a first residual image; Correcting the first residual image according to multiple candidate values ​​of the first scaling factor to determine multiple output images; A target value of the first scaling factor is determined from among the plurality of candidate values ​​according to the plurality of output images.

12. The method according to claim 11, wherein: The multiple candidate values ​​are determined based on the following method: Traversal is performed within the value range of the first scaling factor according to the traversal step size.

13. The method according to claim 12, wherein: The value range of the first scaling factor includes a first value interval and a second value interval, the first value interval corresponds to a first traversal step length, the second value interval corresponds to a second traversal step length, and the first traversal step length is different from the second traversal step length.

14. The method according to claim 12, wherein: The method further comprises: The first information is written into a bitstream, where the first information is used to indicate the number of traversals corresponding to the first scaling factor.

15. The method according to claim 11, wherein: The method further comprises: Determine a second residual image according to the original image and the reconstructed image; determining a target value of a second scaling factor according to the first residual image and the second residual image; According to a target value of the first scaling factor and a target value of the second scaling factor, a target scaling factor is determined from the first scaling factor and the second scaling factor.

16. The method according to claim 15, wherein: The determining, according to a target value of the first scaling factor and a target value of the second scaling factor, a target scaling factor from the first scaling factor and the second scaling factor, includes: Correcting the first residual image according to a target value of the second scaling factor to determine a first output image; The target scaling factor is determined from the first scaling factor and the second scaling factor according to the first output image and the second output image, and the second output image is an output image corresponding to the target value of the first scaling factor.

17. The method according to claim 16, wherein: The determining the target scaling factor from the first scaling factor and the second scaling factor according to the first output image and the second output image includes: Determining a cost corresponding to the first scaling factor according to the first output image and the original image; determining a cost corresponding to the second scaling factor according to the second output image and the original image; The target scaling factor is determined from the first scaling factor and the second scaling factor according to the cost corresponding to the first scaling factor and the cost corresponding to the second scaling factor.

18. The method according to claim 11, wherein: The method further comprises: The first information is written into the bitstream, where the first information is index information of the target value among the multiple candidate values.

19. The method according to claim 11, wherein: The method further comprises: Writing second information into a bitstream, where the second information is used to indicate whether to correct the first residual image based on the first scaling factor.

20. The method according to claim 19, wherein: The second information includes a first value and a second value, the first value indicates that the first residual image is corrected based on the first scaling factor, and the second value indicates that the first residual image is not corrected based on the first scaling factor.

21. The method according to claim 20, wherein: If the value of the second information is the second value, third information is written into the code stream, the third information being a target value of a second scaling factor, the second scaling factor being determined based on the first residual image and the second residual image, and the second residual image being determined based on the original image and the reconstructed image.

22. The method according to claim 11, wherein: The performing loop filtering on the reconstructed image according to the neural network to determine the first residual image includes: Performing loop filtering on the reconstructed image according to the neural network to determine a filtered image; The first residual image is determined according to the filtered image and the reconstructed image.

23. The method according to claim 11, wherein: The step of correcting the first residual image according to the plurality of candidate values ​​of the first scaling factor to determine the plurality of output images comprises: Correcting the first residual image according to multiple candidate values ​​of the first scaling factor to determine multiple corrected residual images; The plurality of output images are determined according to the plurality of modified residual images and the reconstructed image.

24. The method according to claim 11, wherein: The determining, according to the multiple output images, a target value of the first scaling factor from the multiple candidate values ​​comprises: Determining costs corresponding to the multiple candidate values ​​according to the multiple output images and the original image; The target value is determined from the multiple candidate values ​​according to the costs corresponding to the multiple candidate values.

25. The method according to claim 11, wherein: The reconstructed image is a frame of image or an image block in a frame of image.

26. A decoder, comprising: A parsing unit configured to parse the code stream to determine the first information; A filtering unit is configured to determine a target value of the first scaling factor from multiple candidate values ​​of the first scaling factor according to the first information; perform loop filtering on the reconstructed image according to a neural network to determine a first residual image; and correct the first residual image according to the target value of the first scaling factor to determine a reconstructed image after loop filtering.

27. A decoder, comprising: Memory for storing computer programs; A processor, configured to execute the method according to any one of claims 1 to 10 when running the computer program.

28. An encoder, comprising: a determining unit configured to determine a reconstructed image of the original image; A filtering unit configured to perform loop filtering on the reconstructed image according to a neural network to determine a first residual image; The first residual image is corrected according to a plurality of candidate values ​​of the first scaling factor to determine a plurality of output images; and a target value of the first scaling factor is determined from the plurality of candidate values ​​according to the plurality of output images.

29. An encoder, comprising: Memory for storing computer programs; A processor, configured to execute the method according to any one of claims 11 to 25 when running the computer program.

30. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 10 or the method according to any one of claims 11 to 25 is implemented.

Images