Encoding and decoding method and device, encoding equipment, decoding equipment and storage medium

Abstract

The invention discloses a coding and decoding method and device, coding equipment, decoding equipment and a storage medium, and the decoding method comprises the steps: decoding a code stream, and determining related syntax elements of a current coding tree unit; according to the related syntax elements, determining a target loop filtering model of the current coding tree unit from candidate loop filtering models based on a neural network; determining reference sample information of the current coding tree unit; and inputting the reference sample information of the current coding tree unit into the target loop filtering model for filtering, and outputting reconstructed sample information after filtering. A coding end uses a candidate loop filtering module to filter a coding tree unit, calculates a distortion cost value, determines a target loop filtering model corresponding to the minimum distortion cost value and writes related syntax elements into a code stream, and a decoding end only needs to analyze the code stream and selects an optimal target loop filtering model for the current coding tree unit according to the related syntax elements. Therefore, the filtering performance of the reconstructed sample of the current coding tree unit is improved.

Description

Coding and decoding method, device, encoding device, decoding device and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of video coding and decoding technology, and in particular to a coding and decoding method, apparatus, encoding device, decoding device, and storage medium. Background Art

[0002] As demand for video display quality increases, new video applications such as high-definition and ultra-high-definition video have emerged. The Joint Video Exploration Team (JVET) of the International Organization for Standardization (ISO / IEC) and ITU-T has developed the next-generation video coding standard H.266 / Versatile Video Coding (VVC).

[0003] Neural networks have been introduced into the field of video coding and decoding. Leveraging the powerful learning capabilities of neural networks, coding and decoding tools based on neural networks often have very high coding and decoding efficiency. For example, there are intra-frame prediction methods based on neural networks, inter-frame prediction methods based on neural networks, and loop filtering methods based on neural networks. Among them, the coding performance of loop filtering methods based on neural networks is the most outstanding. However, the current loop filtering methods based on neural networks do not fully utilize the advantages of neural network models. In some coding and decoding scenarios, the loop filtering methods based on neural networks do not significantly improve the filtering effect, and may even reduce the filtering efficiency. Therefore, the loop filtering methods based on neural networks need to be optimized.

[0004] Summary of the Invention

[0005] Embodiments of the present invention provide a coding and decoding method, apparatus, coding device, decoding device, and storage medium.

[0006] In a first aspect, an embodiment of the present application provides a decoding method, comprising:

[0007] Decode the code stream and determine the relevant syntax elements of the current coding tree unit;

[0008] Determining a target loop filter model for a current coding tree unit from candidate loop filter models based on a neural network according to the relevant syntax elements;

[0009] Determining reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0010] The reference sample information of the current coding tree unit is input into the target loop filter model for filtering, and the filtered reconstructed sample information is output.

[0011] In a second aspect, an embodiment of the present application provides an encoding method, including:

[0012] Determining reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0013] Input the reference sample information of the current coding tree unit into the candidate loop filter model based on the neural network for filtering, and output the filtered reconstructed sample information;

[0014] Determining a first distortion cost value of the current coding tree unit according to original sample information of the current coding tree unit and the filtered reconstructed sample information;

[0015] Determining a target loop filter model for the current coding tree unit from candidate loop filter models based on a neural network according to a first distortion cost value of the current coding tree unit;

[0016] Encode the relevant syntax elements of the target loop filter model of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0017] In a third aspect, an embodiment of the present application provides a decoding device, comprising:

[0018] A decoding unit configured to decode the code stream and determine relevant syntax elements of a current coding tree unit;

[0019] A first determining unit is configured to determine a target loop filter model of a current coding tree unit from candidate loop filter models based on a neural network according to the relevant syntax elements;

[0020] A second determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0021] The filtering unit is configured to input the reference sample information of the current coding tree unit into the first loop filtering model for filtering, and output the filtered reconstructed sample information.

[0022] In a fourth aspect, an embodiment of the present application provides an encoding device, comprising:

[0023] A first determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0024] A filtering unit configured to input the reference sample information of the current coding tree unit into a candidate loop filter model based on a neural network for filtering, and output the filtered reconstructed sample information;

[0025] a second determining unit configured to determine a first distortion cost value of the current coding tree unit based on original sample information of the current coding tree unit and the filtered reconstructed sample information; and determine a target loop filter model for the current coding tree unit from candidate loop filter models based on a neural network based on the first distortion cost value of the current coding tree unit;

[0026] The coding unit is configured to encode relevant syntax elements of the target loop filter model of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0027] In a fifth aspect, an embodiment of the present application further provides a decoding device, comprising: a first memory and a first processor; the first memory stores a computer program that can be run on the first processor, and the first processor implements the decoding method of the decoder when executing the program.

[0028] In a sixth aspect, an embodiment of the present application further provides an encoding device, comprising: a second memory and a second processor; the second memory stores a computer program that can be run on the second processor, and the second processor implements the encoding method of the encoder when executing the program.

[0029] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, comprising: a computer program stored thereon, which, when executed by a first processor, implements the decoding method of the decoder; or, when executed by a second processor, implements the encoding method of the encoder.

[0030] Using the above technical solution, the encoder uses candidate loop filter modules to filter the coding tree unit, calculate the distortion cost, determine the target loop filter model corresponding to the minimum distortion cost, and write the relevant syntax elements into the bitstream. The decoder only needs to parse the bitstream and select the optimal target loop filter model for the current coding tree unit based on the relevant syntax elements, thereby improving the filtering performance of the reconstructed samples of the current coding tree unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is a block diagram of an encoder according to an embodiment of the present application;

[0032] FIG2 is a block diagram of a decoder according to an embodiment of the present application;

[0033] FIG3 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0034] FIG4 is a schematic diagram of a first neural network model provided in an embodiment of the present application;

[0035] FIG5 is a schematic diagram of an attention residual block provided in an embodiment of the present application;

[0036] FIG6 is a schematic diagram of a second neural network model provided in an embodiment of the present application;

[0037] FIG7 is a schematic diagram of a residual block provided in an embodiment of the present application;

[0038] FIG8 is a schematic diagram of a third neural network model provided in an embodiment of the present application;

[0039] FIG9 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0040] FIG10 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present application;

[0041] FIG11 is a schematic diagram of a specific hardware structure of a decoding device provided in an embodiment of the present application;

[0042] FIG12 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application;

[0043] FIG13 is a schematic diagram of a specific hardware structure of an encoding device provided in an embodiment of the present application;

[0044] FIG14 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present invention, the implementation of the embodiments of the present invention is described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference only and are not intended to limit the embodiments of the present invention.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0047] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first, second, and third" in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequential order where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0049] Referring to Figure 1, which shows a schematic block diagram of the composition of an encoder provided in an embodiment of the present application. As shown in Figure 1, the encoder (specifically, a "video encoder") 100 may include a transform and quantization unit 101, an intra-frame estimation unit 102, an intra-frame prediction unit 103, a motion compensation unit 104, a motion estimation unit 105, an inverse transform and inverse quantization unit 106, a filter control analysis unit 107, a filtering unit 108, an encoding unit 109, and a decoded image cache unit 110, etc., wherein the filtering unit 108 can implement deblocking filtering and sample adaptive offset (SAO) filtering, and the encoding unit 109 can implement header information encoding and context-based adaptive binary arithmetic coding (CABAC).For the input original video signal, a video coding block can be obtained by dividing it into coding tree blocks (CTUs). Then, the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transformation and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to perform intra-frame prediction on the video coding block; specifically, the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to determine the intra-frame prediction mode to be used to encode the video coding block; the motion compensation unit 104 and the motion estimation unit 105 are used to perform inter-frame prediction coding of the received video coding block relative to one or more blocks in one or more reference images to provide temporal prediction information; the motion estimation performed by the motion estimation unit 105 is a process of generating a motion vector, which can estimate the motion of the video coding block, and then the motion compensation unit 104 calculates the motion vector based on the motion determined by the motion estimation unit 105. vector performs motion compensation; after determining the intra-frame prediction mode, the intra-frame prediction unit 103 is also used to provide the selected intra-frame prediction data to the encoding unit 109, and the motion estimation unit 105 also sends the calculated motion vector data to the encoding unit 109; in addition, the inverse transform and inverse quantization unit 106 is used to reconstruct the video coding block, reconstruct the residual block in the pixel domain, and remove the block effect artifacts of the reconstructed residual block through the filter control analysis unit 107 and the filtering unit 108. The reconstructed residual block is then added to a predictive block in the frame of the decoded image buffer unit 110 to generate a reconstructed video coding block; the encoding unit 109 is used to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-frame prediction mode, and output the code stream of the video signal; and the decoded image buffer unit 110 is used to store the reconstructed video coding block for prediction reference. As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image buffer unit 110 .

[0050] Referring to Figure 2, which shows a schematic block diagram of a decoder provided in an embodiment of the present application, as shown in Figure 2, the decoder (specifically, a "video decoder") 200 includes a decoding unit 201, an inverse transform and inverse quantization unit 202, an intra-frame prediction unit 203, a motion compensation unit 204, a filtering unit 205, and a decoded image buffer unit 206. The decoding unit 201 can implement header information decoding and CABAC decoding, and the filtering unit 205 can implement deblocking filtering and SAO filtering. After the input video signal is encoded as shown in FIG1 , a code stream of the video signal is output; the code stream is input to the decoder 200 and first passes through the decoding unit 201 to obtain the decoded transform coefficients; the transform coefficients are processed by the inverse transform and inverse quantization unit 202 to generate a residual block in the pixel domain; the intra-frame prediction unit 203 can be used to generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and the data from the previously decoded block of the current image or picture; the motion compensation unit 204 determines the prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses The prediction information is used to generate a predictive block for the video decoding block being decoded; a decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 202 with the corresponding predictive block generated by the intra-frame prediction unit 203 or the motion compensation unit 204; the decoded video signal passes through the filtering unit 205 to remove blocking artifacts, thereby improving video quality; the decoded video block is then stored in the decoded image buffer unit 206, which stores reference images used for subsequent intra-frame prediction or motion compensation, and is also used for outputting the video signal, thereby obtaining the restored original video signal.

[0051] It should be noted that the method of the embodiment of the present application is mainly applied to the filtering unit 108 shown in Figure 1 and the filtering unit 205 shown in Figure 2. In other words, the embodiment of the present application can be applied to both the encoder and the decoder, or even to both the encoder and the decoder at the same time, but the embodiment of the present application is not specifically limited thereto.

[0052] In one embodiment of the present application, referring to FIG3 , a flowchart of a decoding method provided by an embodiment of the present application is shown. As shown in FIG3 , the method may include:

[0053] Step 301: Decode the code stream and determine the relevant syntax elements of the current coding tree unit;

[0054] Step 302: Determine a target in-loop filter model for a current coding tree unit from candidate in-loop filter models based on a neural network according to the relevant syntax elements;

[0055] The relevant syntax elements are used to indicate the target loop filter model of the coding tree unit. The relevant syntax elements include one or more of the sequence level syntax elements, picture level syntax elements, slice level syntax elements and coding tree unit level syntax elements.

[0056] Exemplarily, in some embodiments, the relevant syntax element includes a first syntax element. According to the first syntax element, a target loop filter model of the current coding tree unit is determined from candidate loop filter models based on a neural network.

[0057] In some embodiments, the first syntax element includes one of the following: a first syntax element at an image sequence level, used to indicate a target loop filtering model for all coding tree units in an image sequence; a first syntax element at an image level, used to indicate a target loop filtering model for all coding tree units in an image; a first syntax element at a slice level, used to indicate a target loop filtering model for all coding tree units in a slice (Slice, also called a "slice"); a first syntax element at a coding tree unit level, used to indicate a target loop filtering model for a coding tree unit.

[0058] Exemplarily, in some embodiments, the relevant syntax elements further include a second syntax element. The method further includes: determining, based on the second syntax element, whether the current image block where the current coding tree unit is located is allowed to use a second loop filter model based on a neural network; wherein the second loop filter model is a candidate loop filter model; when, based on the second syntax element, it is determined that the current image block where the current coding tree unit is located uses the second loop filter model based on a neural network, determining, based on the first syntax element, a target loop filter model for the current coding tree unit from the candidate loop filter models based on a neural network. Determining, based on the second syntax element, that the second loop filter model is allowed to be used, further determining, based on the first syntax element, the target loop filter model; determining, based on the second syntax element, that the second loop filter model is not allowed to be used, further determining to use a preset first loop filter model, or determining not to use a loop filter technology based on a neural network, or determining to use another filtering technology. wherein the first loop filter model is a candidate loop filter model, or is not a candidate loop filter model.

[0059] Exemplarily, the current image block includes at least one of the following: an image sequence where the current coding tree unit is located, an image where the current coding tree unit is located, a slice where the current coding tree unit is located, and a current image tree unit.

[0060] Exemplarily, the candidate loop filter models include a first loop filter model and a second loop filter model. The first loop filter model can be understood as an original loop filter model, and the second loop filter model can be understood as a replacement loop filter model for the original loop filter model. If the second loop filter model is determined to be permitted, it indicates that any loop filter model can be selected from the candidate loop filter models. If the second loop filter model is determined not to be permitted, it indicates that only the first loop filter model can be used.

[0061] Exemplarily, the second syntax element includes at least one of the following: a second syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use the second loop filtering model based on the neural network; a second syntax element at the image level, used to indicate whether the image is allowed to use the second loop filtering model based on the neural network; a second syntax element at the slice level, used to indicate whether the slice is allowed to use the second loop filtering model based on the neural network; a second syntax element at the coding tree unit level, used to indicate whether the coding tree unit is allowed to use the second loop filtering model based on the neural network.

[0062] In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level. In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level and a second syntax element at a picture level. In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level and a second syntax element at a slice level. In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level, a second syntax element at a picture level (or a slice level), and a second syntax element at a coding tree unit level.

[0063] Exemplarily, in some embodiments, the relevant syntax elements further include a third syntax element, and based on the third syntax element, it is determined whether the current image block where the current coding tree unit is located uses a neural network-based loop filtering technology; when it is determined based on the third syntax element that the current image block where the current coding tree unit is located uses a neural network-based loop filtering technology, the target loop filtering model for the current coding tree unit is determined from candidate loop filtering models based on a neural network based on the first syntax element. In some embodiments, when the third syntax element is a first preset value, it is determined that all coding tree units in the current image block do not use a neural network-based loop filtering technology; when the third syntax element is a second preset value, it is determined that all coding tree units in the current image block use a neural network-based loop filtering technology; when the third syntax element is a second preset value, it is determined that some coding tree units in the current image block use a neural network-based loop filtering technology.

[0064] Exemplarily, in some embodiments, the third syntax element includes at least one of the following: a third syntax element at the image sequence level, used to indicate whether the image sequence uses a neural network-based loop filtering technology; a third syntax element at the image level, used to indicate whether the image uses a neural network-based loop filtering technology; a third syntax element at the slice level, used to indicate whether the slice uses a neural network-based loop filtering technology; a third syntax element at the coding tree unit level, used to indicate whether the coding tree unit uses a neural network-based loop filtering technology.

[0065] In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a picture level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a slice level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level, a third syntax element at a picture level (or a slice level), and a third syntax element at a coding tree unit level.

[0066] In some exemplary embodiments, the relevant syntax elements further include a fourth syntax element, and the image type or slice type of the current coding tree unit is determined based on the fourth syntax element. When the image type or slice type of the current coding tree unit is determined to be a preset type based on the fourth syntax element, the target loop filter model for the current coding tree unit is determined from candidate loop filter models based on a neural network based on the first syntax element. In other words, the candidate loop filter model based on a neural network is only permitted for loop filtering when the image type or slice type of the current coding tree unit is the preset type; otherwise, it is not permitted. Different models can be used for different image or slice types, such as intra I_Slice and dual reference image B_Slice, and the input components may differ. Specifically, the I_Slice model may include an additional partition information "partition" as input. Different color components may also have different applicable models, and the input information may also differ. For example, the chroma component model typically requires input of not only reconstructed sample information (rec) for the chroma component, but also reconstructed sample information (rec) for the luma component to improve filtering performance.

[0067] In some embodiments, the image type or slice type is B_slice, and according to the first syntax element, a target loop filter model of the current coding tree unit is determined from candidate loop filter models based on a neural network.

[0068] In some embodiments, the fourth syntax element includes one of the following: a fourth syntax element at the picture sequence level, used to indicate the picture type or slice type of all coding tree units in the picture sequence; a fourth syntax element at the picture level, used to indicate the picture type or slice type of all coding tree units in the picture; a fourth syntax element at the slice level, used to indicate the picture type or slice type of all coding tree units in the slice; a fourth syntax element at the coding tree unit level, used to indicate the picture type or slice type of the coding tree unit.

[0069] In some embodiments, the relevant syntax elements further include a fifth syntax element. Based on the fifth syntax element, it is determined whether the current image block allows the use of a neural network-based loop filtering technique. Based on the fifth syntax element, it is determined that the use of a neural network-based loop filtering technique is allowed, and subsequent relevant syntax elements are parsed to determine a target loop filtering model. Based on the fifth syntax element, it is determined that the use of a neural network-based loop filtering technique is not allowed, and that other filtering techniques are used, or that no filtering technique is used.

[0070] In some embodiments, the fifth syntax element includes at least one of the following: a fifth syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the image level, used to indicate whether the image is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the slice level, used to indicate whether the slice is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the coding tree unit level, used to indicate whether the coding tree unit is allowed to use the neural network-based loop filtering technology.

[0071] In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a picture level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a slice level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level, a fifth syntax element at a picture level (or a slice level), and a fifth syntax element at a coding tree unit level.

[0072] In summary, when determining that the current image block uses a candidate loop filter model based on a neural network according to at least one of the second syntax element, the third syntax element, the fourth syntax element, and the fifth syntax element, the target loop filter model is determined from the candidate loop filter models according to the first syntax element.

[0073] Step 303: Determine reference sample information of the current coding tree unit; wherein the reference sample information at least includes: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0074] For example, as shown in FIG1 , each image in the input video is divided into square largest coding units (LCUs) of the same size (e.g., 128x128, 64x64, etc.). Each LCU can be divided into rectangular coding units (CUs) according to a rule. Coding units may also be divided into prediction units (PUs), transform units (TUs), etc. The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra prediction and inter prediction. Inter prediction includes motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in an image of a video, the intra prediction method is used in video coding and decoding technology to eliminate spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent images in a video, the inter prediction method is used in video coding and decoding technology to eliminate temporal redundancy between adjacent frames, thereby improving coding and decoding efficiency.

[0075] Intra-frame prediction or inter-frame prediction is used for the current block to generate a prediction block for the current block. This prediction block is then used to form a prediction block (i.e., prediction sample information) for the coding tree unit. The bitstream is then parsed to obtain a quantization coefficient matrix, which is then dequantized and inversely transformed to obtain a residual block. The prediction block and residual block are then added together to form a reconstructed block. The reconstructed block forms the reconstructed image (i.e., reconstructed sample information) for the coding tree unit. The reconstructed image is then loop-filtered using the coding tree unit (i.e., the maximum coding unit size) as the basic processing unit to produce the decoded image.

[0076] In some embodiments, the reconstructed sample information includes reconstructed sample information of a first color component and reconstructed sample information of a second color component of the current coding tree unit, and the predicted sample information includes predicted sample information of the first color component and predicted sample information of the second color component of the current coding tree unit. Specifically, when loop filtering is performed on chroma components, the loop filtering model of the chroma components generally requires input of not only the reconstructed sample information rec of the chroma components, but also the reconstructed sample information rec of the luminance component to improve filtering performance.

[0077] Exemplarily, in some embodiments, the current image block includes at least one of the following: an image sequence where the current coding tree unit is located, an image where the current coding tree unit is located, a slice where the current coding tree unit is located, and a current image tree unit.

[0078] Step 304: input the reference sample information of the current coding tree unit into the target loop filter model for filtering, and output the filtered reconstructed sample information.

[0079] In some embodiments, the parameter sample information further includes: a quantization parameter; and the method further includes:

[0080] Determining, based on the sixth syntax element, whether to adjust a quantization parameter of a current image block where the current coding tree unit is located; when determining, based on the sixth syntax element, that the quantization parameter of the current image block where the current coding tree unit is located is to be adjusted, determining, based on the seventh syntax element, the adjusted quantization parameter of the current image block where the current coding tree unit is located; and inputting the adjusted quantization parameter into the target loop filter model.

[0081] In some embodiments, the sixth syntax element includes at least one of the following: a sixth syntax element at the image sequence level, used to indicate whether to adjust the quantization parameters of the coding tree units in the image sequence; a sixth syntax element at the image level, used to indicate whether to adjust the quantization parameters of the coding tree units in the image; a sixth syntax element at the slice level, used to indicate whether to adjust the quantization parameters of the coding tree units in the slice; a sixth syntax element at the coding tree unit level, used to indicate whether to adjust the quantization parameters of the coding tree units.

[0082] In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a picture level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a slice level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level, a sixth syntax element at a picture level (or a slice level), and a sixth syntax element at a coding tree unit level.

[0083] The seventh syntax element includes one of the following: a seventh syntax element at the picture sequence level, used to indicate the adjusted quantization parameters of all coding tree units in the picture sequence; a seventh syntax element at the picture level, used to indicate the adjusted quantization parameters of all coding tree units in the picture; a seventh syntax element at the slice level, used to indicate the adjusted quantization parameters of all coding tree units in the slice; or a seventh syntax element at the coding tree unit level, used to indicate the adjusted quantization parameters of the coding tree unit.

[0084] In some embodiments, the reference sample information also includes at least one of the following: quantization parameters, boundary strength information of the current coding tree unit, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image, the slice type of the current coding tree unit and the partitioning information of the current coding tree unit.

[0085] In actual applications, the inputs of different loop filter models may not be exactly the same. The target loop filter model and the input information of the target loop filter model are determined based on relevant syntax elements. In some embodiments, the candidate loop filter models include a first loop filter model and a second loop filter model; wherein the reference sample information input to the first loop filter model also includes: quantization parameters and boundary strength information of the current coding tree unit; the reference sample information input to the second loop filter model also includes: quantization parameters, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the forward reference image, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the backward reference image.

[0086] Correspondingly, in some embodiments, the method further includes: when the target loop filtering model is the second loop filtering model, obtaining the first reference image of the first reference image list, and obtaining the first reference image of the second reference image list; when the first reference image of the first reference image list and the first reference image of the second reference image list are the same image, obtaining the second reference image of the first reference image list or the second reference image list; using the reference image of the first reference image list obtained as the forward reference image, and the reference image of the first reference image list as the backward reference image.

[0087] For the input of the second loop filter model with multiple reference images, a duplicate check can be performed on the first and second reference images to avoid inputting the same two coding tree unit information. If the two reference images are the same, the second candidate image in the second reference image list can be used as the input for the second reference image; the third candidate image in the second reference image list can also be used as the input for the second reference image; similarly, the second candidate image in the first reference image list can also be used as the input for the first reference image. This article will not go into details here.

[0088] Exemplarily, in some embodiments, the method further includes: when the time domain level of the current coding tree unit is greater than or equal to the time domain level threshold, determining the target loop filter model of the current coding tree unit as the second loop filter model; when the time domain level of the current coding tree unit is less than the time domain level threshold, determining the target loop filter model of the current coding tree unit from the candidate loop filter models based on the neural network according to the relevant syntax elements.

[0089] For example, in some embodiments, the current coding tree unit is a maximum coding unit, or is obtained by rescaling the maximum coding unit. That is, patchWidth and patchHeight are the block sizes input to the model, which can be the coding tree unit size or a larger patch block after padding the coding tree unit.

[0090] Exploration of neural network-based loop filtering solutions primarily focuses on two forms: a multi-model switchable solution and a single-model solution. These two solutions serve as baseline solutions for neural network loop filtering. The reference software integrating these two solutions will be named the Neural Network Based Video Coding Reference Software (NNVC). All neural network-related proposals must use NNVC as a benchmark for performance and complexity comparison. It is worth noting that the basic processing unit of the neural network loop filtering solution is the coding tree unit, which is the maximum coding unit size.

[0091] The biggest difference between the first multi-model switchable solution and the second single-model solution is that the multi-model solution can provide better performance than a single model for different application scenarios or configuration conditions; however, the most obvious disadvantage of the multi-model solution is that it requires storing multiple models and requires loading a large number of models during calculation and inference.

[0092] The specific first multi-model solution is shown in Figure 4. The neural network-based loop filter model framework mainly consists of the input part, the main network part, and the output part. The main reasoning is shown in the figure as a 3x3 convolution connected to multiple AR blocks, and finally a 3x3 convolution operation and a data position shuffle operation. The 2Nx2N input is usually the size of a coding tree unit, but in some specific implementations or methods, the boundary pixels are often copied outward, commonly known as the padding operation. This is because, as shown in the figure above, if the 3x3 convolutional neural network needs to perform a convolution operation on the boundary, it needs to expand outward, otherwise the convolution cannot be performed. It is worth noting that the design of the attention residual block (Attention Residual block) shown in Figure 5 may also be different in different implementations and methods. The numbers and designs here are only for the convenience of understanding the entire neural network-based loop filter model framework.

[0093] The Attention Residual block is a modular module consisting of multiple convolutional layers, a Prelude (PReLU) activation layer, and an Attention layer. This article does not provide a detailed description of the various operational layers within the Attention Residual block. The use, number, and internal structure of these modular modules do not significantly impact the proposed techniques. It should be noted that the proposed solution applies to both the Attention Residual block shown in Figure 5 and other ResBlocks.

[0094] The input part currently mainly includes reconstructed sample data rec, predicted sample data pred, quantization parameter information QP, boundary strength information BS, etc. Different slice types, such as intra-frame I_Slice and dual reference image B_Slice, can use different models, and the input part may be different. Specifically, the I_Slice model can have an additional partition information partition as input. Different color components may have different applicable models, and the input information may also be different. Specifically, the chroma component model usually requires not only the reconstructed sample data rec of the chroma component, but also the reconstructed sample data rec of the luminance component to improve filtering performance.

[0095] As for the output part output, this part is basically the residual information res of the current coding tree unit or the reconstructed sample information rec of the current coding tree unit.

[0096] As mentioned above, for different filtering targets, such as luma and chroma, a multi-model solution can train a separate model for luma and a separate model for chroma. For example, for I_slice and B_slice, a multi-model solution can train a model for I_slice and a separate model for B_slice. Specifically, the current mainstream solution has four models, corresponding to luma I_slice, luma B_slice, chroma I_slice, and chroma B_slice.

[0097] The embodiment of the present application also provides another multi-model solution as shown in Figure 6. The neural network loop filter model framework is mainly composed of an input part, a main network part and an output part, wherein the main reasoning is shown in Figure 6 by a 3x3 convolution connected to multiple Resblocks, and finally a 3x3 convolution operation and a shuffle operation on the data transformation position. Figure 6 shows a replacement model for the original B_Slice model shown in Figure 4, called the Alter_B_Slice model. There are four models in the first multi-model solution, including luminance I_Slice, luminance B_Slice, chrominance I_Slice and chrominance B_Slice. The Alter_B_Slice model can replace luminance B_Slice, chrominance B_Slice or both. However, this replacement is not completely for all B_slice type images or slices, but for some images or slices.

[0098] According to existing knowledge, B_Slice type pictures or slices can reference multiple decoded pictures or slices, and in a group of pictures (GOP), B_Slice pictures or slices have different temporal levels depending on the distance between the decoded pictures or slices allowed to be referenced. The Alter_B_Slice model can be used in pictures or slices of higher levels, while pictures or slices of other temporal levels use the original B_Slice model.

[0099] Figure 7 shows a design of a residual block. It should be noted that different implementations and methods may also be different. The numbers and designs here are only for the convenience of understanding the entire neural network-based loop filtering model framework.

[0100] The conditions for using the Alter_B_Slice model shown in Figure 6 and the original B_Slice model shown in Figure 4 are as described above. Furthermore, the use of either model for the current image or slice can be determined based on different temporal layers. Images or slices at higher layers use the Alter_B_Slice model, while images or slices at lower layers use the original B_Slice model. The higher layer can be defined as images or slices with a Temporal Layer ID greater than or equal to 3 (TiD 3), while the lower layer can be defined as any other ID, such as TiD 0, 1, or 2. This is for ease of understanding only; in practice, the temporal layer number can be any number within the standard's permitted range.

[0101] The difference between the Alter_B_Slice model and the original B_Slice model lies in the input part. By comparing with the original B_Slice model, it can be found that the Alter_B_Slice model removes the boundary strength information BS, retains the reconstructed sample rec, the predicted sample pred, and the quantization parameter information QP. In addition, the Alter_B_Slice model also adds a forward reference image Forw and a backward reference image Bacw as input information of the model to increase the filtering capability of the neural network loop filter model at the time domain level. Here, the forward reference image and the backward reference image can be defined as being obtained from two reference image lists allowed by the standard. For example, the forward reference image can be the first reference image in the first reference image list, and the backward reference image can be the first reference image in the second reference image list.

[0102] Exemplarily, the candidate loop filter models include a first loop filter model and a second loop filter model. The first loop filter model can be an original loop filter model B_Slice model, and the second loop filter model can be understood as an alternative loop filter model Alter_B_Slice model. When it is determined that the Alter_B_Slice model is allowed to be used, the target loop filter model is further determined based on relevant syntax elements. When it is determined that the Alter_B_Slice model is not allowed to be used, other loop filter models are further determined based on relevant syntax elements, or it is determined to use other filtering technologies.

[0103] In some embodiments, the candidate loop filter models may further include an I_Slice model. In some embodiments, the I_Slice model specifically includes a luma I_Slice model and a chroma I_Slice model.

[0104] In some embodiments, the candidate loop filter model may also include a single model. The embodiment of the present application also provides a single model solution as shown in Figure 8. The input and filter size of the single model solution are the same as those of the two multi-model solutions mentioned above, but it is obvious that the combination block used in this solution is different from that in the previous two solutions. The combination block used here is ResBlock. As shown in Figure 8, ResBlock is mainly composed of a 1x1xKxK Conv. convolution layer, a ReLU activation layer connected to a 1x1xKxK Conv. convolution layer and a 3x3xKxK Conv. convolution layer. It should be pointed out that both the ARblock of the first multi-model solution and the ResBlock of the second multi-model solution and the single model solution have jump connections, that is, the input and output are connected in this combination block.

[0105] For the input part, the input types of the single model solution mainly consist of reconstructed samples rec, predicted samples pred, and three constant input base quantization parameter information BaseQP, slice-level quantization parameter information SliceQP, and slice type Slicetype. Here, "slice" Slice can be broadly understood as image level or picture level. Since a single model can process different color components and different image types, the input part needs to input this information at one time to help the neural network better filter the current coding tree unit. Therefore, the reconstructed samples rec include luminance reconstructed samples rec and chrominance reconstructed samples rec, the predicted samples pred include luminance prediction samples pred and chrominance prediction samples pred, and the slice type Slicetype indicates that the current coding tree unit is of I_Slice type, B_Slice type, or even other types.

[0106] For the output part, if it is a luma model, the output is luma reconstructed samples rec or luma residual information res, and if it is a chroma model, the output is chroma reconstructed samples rec or chroma residual information res. For the single model solution, the output directly contains luma and chroma reconstructed samples rec or residual information res.

[0107] In summary, the main differences in the main frameworks of the three models lie in the input part and the number of models. This application does not make specific restrictions on the main part of the neural network model. In addition to the above three structures, depth-separable convolutions can also be used to replace the above convolution operations.

[0108] During model training and usage, due to the generalization of the model and the inclusion of quantization parameter information in the model training input, different filtering results can be obtained by adjusting these input parameters when the filtered coding tree units are obtained through model inference. The encoder selects the optimal parameters for these different filtering results based on the principle of minimizing rate distortion. These parameters are written into the bitstream and transmitted at the image level, slice level, or coding tree unit level (CTU). The decoder parses the bitstream to obtain these adjusted parameter information and performs the same adjustments as the encoder, resulting in the same filtering results as the encoder. This section can be summarized as model input parameter adjustment.

[0109] In addition, a scaling operation scale is usually performed on the reconstructed sample rec after filtering. The scaling factor scaling factor is obtained at the encoding end by the global minimum mean square error method or other methods that calculate a certain variance to minimize the difference between the original sample, the reconstructed sample after neural network filtering, and the reconstructed sample before filtering. The scaling factor is written into the bitstream at the slice level, the coding tree unit level CTU, or other combined division areas and transmitted to the decoding end. The decoding end obtains the same scaling factor as the encoding end by parsing the bitstream and scales the reconstructed sample rec after filtering to obtain the final reconstructed sample rec of the coding tree unit. The specific scaling operation is as follows: rec refine =(rec cnn -rec before )*scale_factor+rec before

[0110] Among them, rec refine is the reconstructed sample after scaling, rec cnn is the reconstructed sample after neural network filtering, rec before is the reconstructed sample before neural network filtering, and scale_factor is the scaling factor. It is worth noting that the above is a theoretical calculation process, and the specific implementation includes using some shift operations to replace multiplication operations.

[0111] The decoding method is further illustrated below with examples. The decoding end parses the sequence-level flag sps_nnlf_enable_flag (i.e., the fifth syntax element). If sps_nnlf_enable_flag is true, it indicates that the current bitstream allows the use of neural network-based loop filtering technology, and the subsequent decoding process needs to parse the relevant syntax elements; otherwise, it indicates that the current bitstream does not allow the use of neural network-based loop filtering technology, and the subsequent decoding process does not need to parse the relevant syntax elements. The default relevant syntax elements are initial values ​​or false states.

[0112] If sps_nnlf_enable_flag is true, the sps_nnlf_alter_enable_flag flag related to the technology proposed in this article is parsed; otherwise, sps_nnlf_alter_enable_flag is set to false; if there is no sps_nnlf_enable_flag related syntax element information in the bitstream, sps_nnlf_enable_flag is also set to false by default.

[0113] 1. If sps_nnlf_enable_flag is true, the decoder parses the syntax elements related to the neural network loop filtering technology of the current image or slice, and obtains the image or slice-level identification bit slice_nnlf_flag based on the neural network loop filtering technology; otherwise, all the neural network loop filtering identification positions related to the technology indicated by the sequence level are set to the default values, and step 3 is executed.

[0114] If sps_nnlf_alter_enable_flag is true (i.e., determined according to the value of the second syntax element) and the current image or slice type is B_Slice (i.e., determined according to the value of the fourth syntax element) and slice_nnlf_flag is not 0 (i.e., determined according to the value of the third syntax element), the decoder parses the relevant syntax elements of the technology proposed in this article at the current image or slice level to obtain the image or slice level flag slice_Alter_B_flag (i.e., the first syntax element); otherwise, the image or slice level flag slice_Alter_B_flag is set to false; if there is no slice_Alter_B_flag related syntax element code stream information in the code stream, the slice_Alter_B_flag flag defaults to false.

[0115] 2. If slice_nnlf_flag is 0, it means that all coding tree units in the current image or slice do not use neural network-based loop filtering technology, and execute step 3;

[0116] If slice_nnlf_flag is 1, it indicates that all coding tree units in the current image or slice are filtered using neural network-based loop filtering technology, and the usage flag ctb_nnlf_flag of all coding tree units in the current image or slice is set to true. If the current image or slice type is B_Slice, the current image or slice needs to indicate the loaded model according to slice_Alter_B_flag; otherwise, it can be loaded according to the original rules. Use the loaded neural network loop filter model to filter all coding tree units, where if the model is the Alter_B_Slice model, the reconstructed sample rec, the predicted sample pred, the quantization parameter QP, the reconstructed sample Forw at the corresponding position in the first reference image, and the reconstructed sample Bacw at the corresponding position in the second reference image of the current coding tree unit are input into the Alter_B_Slice model for reasoning to obtain the filtered reconstructed sample of the current coding tree unit; if the model is the original B_Slice model, the reconstructed sample rec, the predicted sample pred, the quantization parameter QP and the boundary strength BS of the current coding tree unit are input into the original B_Slice model for reasoning to obtain the filtered reconstructed sample of the current coding tree unit. If it is other models, the specified relevant information is input into the model for reasoning to obtain the filtered reconstructed sample, which is not elaborated in detail here. Filter all coding tree units in the current image or slice;

[0117] If slice_nnlf_flag is 2, it means that some coding tree units in the current image or slice use neural network-based loop filtering technology, and some coding tree units do not use neural network-based loop filtering technology. It is necessary to further analyze the coding tree unit level usage identification bit ctb_nnlf_flag of all coding tree units in the current image or slice. If the current image or slice type is B_Slice, the current image or slice needs to indicate the loaded model according to slice_Alter_B_flag, otherwise it can be loaded according to the original rules. Traverse all the coding tree units in the current image or slice. If ctb_nnlf_flag is true, the loaded model is used to filter the coding tree unit; if ctb_nnlf_flag is false, the coding unit is not filtered. The filtering operation on the coding tree unit is the same as described above and will not be repeated here.

[0118] After traversing all coding tree units in the current image or slice, the neural network-based loop filtering module ends.

[0119] 3. The decoder continues to traverse other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article and will not be explained in detail here. The decoding end analysis process is shown in Table 1.

[0120] Table 1 is a brief description of the decoding end analysis process.

[0121]

[0122]

[0123] In this embodiment, different quantization parameters can be selected at the coding tree unit level, and the quantization parameters are usually used as constant inputs of the model. This embodiment focuses on combining the adjustment of the quantization parameters with the selection of the model.

[0124] Specifically, the quantization parameter BaseQP of the model input is adjusted according to the preset step size / offset value / compensation value / candidate value. In this embodiment, the compensation value is taken as an example, and the adjustment calculation is as follows: FinalBaseQP=BaseQP+offset

[0125] In the above formula, FinalBaseQP is the quantization parameter BaseQP information finally input into the model, and offset is the quantization parameter compensation value (i.e., adjustment parameter), which can be 0, +5, -5, +10, and -10, etc.

[0126] The decoding end parses the sequence-level flag. If sps_nnlf_enable_flag is true, it means that the current bitstream allows the use of neural network-based loop filtering technology, and the subsequent decoding process needs to parse the relevant syntax elements; otherwise, it means that the current bitstream does not allow the use of neural network-based loop filtering technology, and the subsequent decoding process does not need to parse the relevant syntax elements. The default relevant syntax elements are initial values ​​or false.

[0127] If sps_nnlf_enable_flag is true, the sps_nnlf_alter_enable_flag flag related to the technology proposed in this article is parsed; otherwise, sps_nnlf_alter_enable_flag is set to false; if there is no sps_nnlf_enable_flag related syntax element information in the bitstream, sps_nnlf_enable_flag is also set to false by default.

[0128] 1. If sps_nnlf_enable_flag is true, the decoder parses the syntax elements related to the neural network loop filtering technology of the current image or slice, and obtains the image or slice-level identification bit slice_nnlf_flag based on the neural network loop filtering technology; otherwise, all the neural network loop filtering identification positions related to the technology indicated by the sequence level are set to the default values, and step 3 is executed.

[0129] If sps_nnlf_alter_enable_flag is true and the current image or slice type is B_Slice and slice_nnlf_flag is not 0, the decoder parses the relevant syntax elements of the current image or slice level technology proposed in this article to obtain the image or slice level flag slice_Alter_B_flag; otherwise, the image or slice level flag slice_Alter_B_flag is set to false; if there is no slice_Alter_B_flag related syntax element code stream information in the code stream, the slice_Alter_B_flag flag defaults to false.

[0130] 2. If slice_nnlf_flag is 0, it means that all coding tree units in the current image or slice do not use neural network-based loop filtering technology, and execute step 3;

[0131] If slice_nnlf_flag is 1, it indicates that all coding tree units in the current image or slice are filtered using neural network-based loop filtering technology. The image-level or slice-level neural network loop filter input adjustment parameter slice_nnlf_param in the bitstream is parsed to indicate whether the input quantization parameter information is modified, and the usage flag ctb_nnlf_flag of all coding tree units in the current image or slice is set to true. If the current image or slice type is B_Slice, the current image or slice needs to indicate the loaded model according to slice_Alter_B_flag; otherwise, it can be loaded according to the original rules. Use the loaded neural network loop filter model to filter all coding tree units; wherein, if the model is the Alter_B_Slice model, the reconstructed sample rec, the predicted sample pred, the quantization parameter QP (adjusted according to the seventh syntax element slice_nnlf_param at the image level or slice level), the reconstructed sample information Forw at the corresponding position in the first reference image, and the reconstructed sample information Bacw at the corresponding position in the second reference image of the current coding tree unit are input into the Alter_B_Slice model for reasoning to obtain the filtered reconstructed sample of the current coding tree unit. If the model is the original B_Slice model, the reconstructed sample rec, the predicted sample pred, the quantization parameter QP (adjusted according to the slice_nnlf_param) and the boundary strength BS of the current coding tree unit are input into the original B_Slice model for reasoning to obtain the filtered reconstructed sample of the current coding tree unit. If it is other models, the specified relevant information is input into the model for reasoning to obtain the filtered reconstructed sample, which is not elaborated in detail here. Filter all coding tree units in the current image or slice;

[0132] If slice_nnlf_flag is 2, it indicates that some CTUs in the current image or slice use neural network-based loop filtering technology, while others do not. It is necessary to analyze the CTU-level usage flag ctb_nnlf_flag for all CTUs in the current image or slice and parse the bitstream based on the true or false status of ctb_nnlf_flag. If ctb_nnlf_flag is true for the current CTU, parse the bitstream to obtain ctb_nnlf_param information; otherwise, it is assumed that ctb_nnlf_param does not exist and its variable value is 0. If the current image or slice type is B_Slice, the current image or slice needs to indicate the loaded model according to slice_Alter_B_flag; otherwise, it can be loaded according to the original rules. Traverse all the coding tree units in the current image or slice. If ctb_nnlf_flag is true, use the loaded model to filter the coding tree unit, and adjust the input quantization parameter information according to ctb_nnlf_param. If ctb_nnlf_flag is false, do not filter the coding unit. The filtering operation for the coding tree unit is the same as described above and will not be repeated here.

[0133] In some embodiments, the quantization parameter QP may be further adjusted according to the sixth syntax element slice_nnlf_param_flag and the seventh syntax element slice_nnlf_param at the picture level or slice level, or the quantization parameter QP may be further adjusted according to the sixth syntax element slice_nnlf_param_flag and the seventh syntax element ctb_nnlf_param at the coding tree unit level.

[0134] After traversing all coding tree units in the current image or slice, the neural network-based loop filtering module ends.

[0135] 3. The decoder continues to traverse other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article and is not explained in detail here. The decoding end analysis process is shown in Table 2.

[0136] Table 2 is a brief description of the decoding end analysis process

[0137]

[0138] In some embodiments, in addition to adjusting the quantization parameters, adjustments may also be made to other input components. For example, a geometric transformation may be performed on non-constant parameters such as the input reconstructed sample information and predicted sample information. Exemplary geometric transformations include diagonal flipping, horizontal flipping, vertical flipping, rotating by a preset angle, and the like, or a combination of the two. Specifically, a value of 0 for the seventh syntax element of the adjustment parameter indicates no adjustment is made; a value of 1 indicates an adjustment to the quantization parameters; and a value of 2 indicates a horizontal flip of the reconstructed sample information and the predicted sample information.

[0139] The encoder selects the target in-loop filter model that minimizes the distortion cost. The in-loop filter model usage is identified by a new syntax element and written into the bitstream for transmission to the decoder. The decoder simply parses the bitstream and selects the optimal target in-loop filter model for the current coding tree unit based on the relevant syntax elements.

[0140] An input duplication check operation is performed on a loop filtering model (eg, Alter_B_Slice) with multiple reference images input to ensure that the reference image obtained from the first reference image list and the reference image obtained from the second reference image list are different reference images.

[0141] The model usage is decided according to the time domain level, and coexists with the method of selecting the target loop filter model by the distortion cost value at the encoding end. That is, the high-time domain layer image or slice uses the predetermined second loop filter model (such as Alter_B_Slice) by default, while the non-high-time domain layer image or slice needs to be decided by optimizing the distortion cost value at the encoding end and represented by the relevant syntax elements.

[0142] The present application also provides an encoding method. In one embodiment of the present application, see Figure 9, which shows a schematic flow chart of an encoding method provided by the present application. As shown in Figure 9, the method may include:

[0143] Step 901: Determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0144] Intra-frame prediction or inter-frame prediction is used for the current block to generate a prediction block for the current block. This prediction block is then used to form a prediction block (i.e., prediction sample information) for the coding tree unit. The bitstream is then parsed to obtain a quantization coefficient matrix, which is then dequantized and inversely transformed to obtain a residual block. The prediction block and residual block are then added together to form a reconstructed block. The reconstructed block forms the reconstructed image (i.e., reconstructed sample information) for the coding tree unit. The reconstructed image is then loop-filtered using the coding tree unit (i.e., the maximum coding unit size) as the basic processing unit to produce the decoded image.

[0145] Step 902: inputting the reference sample information of the current coding tree unit into a candidate loop filter model based on a neural network for filtering, and outputting the filtered reconstructed sample information;

[0146] Exemplarily, the reference sample information also includes at least one of the following: quantization parameter, boundary strength information of the current coding tree unit, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image, slice type of the current coding tree unit and partition information of the current coding tree unit.

[0147] In actual applications, the inputs of different loop filter models may not be exactly the same. The target loop filter model and the input information of the target loop filter model are determined based on relevant syntax elements. In some embodiments, the candidate loop filter models include a first loop filter model and a second loop filter model; wherein the reference sample information input to the first loop filter model also includes: quantization parameters and boundary strength information of the current coding tree unit; the reference sample information input to the second loop filter model also includes: quantization parameters, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the forward reference image, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the backward reference image.

[0148] Step 903: determining a first distortion cost value of the current coding tree unit according to the original sample information of the current coding tree unit and the filtered reconstructed sample information;

[0149] Step 904: Determine a target in-loop filter model for the current coding tree unit from candidate in-loop filter models based on a neural network according to the first distortion cost value of the current coding tree unit;

[0150] In some embodiments, determining the target loop filter model for the current coding tree unit from candidate loop filter models based on a neural network based on the first distortion cost value of the current coding tree unit includes: accumulating the first distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the first distortion cost value of the current image block; and determining the target loop filter model for the current coding tree unit based on the first distortion cost value of the current image block corresponding to the candidate loop filter models. Specifically, the loop filter model corresponding to the minimum value of the first distortion cost value is selected as the target loop filter model. In some embodiments, the first distortion cost value may be a rate-distortion cost value.

[0151] Exemplarily, the current image block includes at least one of the following: a current image sequence, a current image, a current slice, and a current coding tree unit. That is, the current image block is the current image sequence, the current image, or the current slice, and the target loop filtering model for the current image block is determined based on the accumulated value of the first distortion cost values ​​of all coding tree units in the current image block. The current image block is the current coding tree unit, and the target loop filtering model for the current coding tree unit is determined based on the first distortion cost value of the current coding tree unit.

[0152] In some embodiments, the method further includes: determining a second distortion cost value of the current coding tree unit based on original sample information of the current coding tree unit and reconstructed sample information of the reference sample information; accumulating the second distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the second distortion cost value of the current image block; and determining whether the current image block uses a neural network-based loop filtering technology based on the minimum distortion cost value of the first distortion cost value and the second distortion cost value of the current image block. In other words, the second distortion cost value without using the neural network-based loop filtering technology and the minimum value of the first distortion cost values ​​corresponding to multiple candidate loop filtering models based on the neural network are calculated, and then the minimum distortion cost value is selected to determine whether the current image block uses a neural network-based loop filtering technology.

[0153] Step 905: Encode the relevant syntax elements of the target loop filter model of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0154] The relevant syntax elements are used to indicate the target loop filter model of the coding tree unit. The relevant syntax elements include one or more of the sequence level syntax elements, picture level syntax elements, slice level syntax elements and coding tree unit level syntax elements.

[0155] Exemplarily, in some embodiments, the relevant syntax element includes a first syntax element. The first syntax element is set according to a target loop filter model of a current coding tree unit.

[0156] In some embodiments, the first syntax element includes one of the following: a first syntax element at an image sequence level, used to indicate a target loop filtering model for all coding tree units in an image sequence; a first syntax element at an image level, used to indicate a target loop filtering model for all coding tree units in an image; a first syntax element at a slice level, used to indicate a target loop filtering model for all coding tree units in a slice; a first syntax element at a coding tree unit level, used to indicate a target loop filtering model for a coding tree unit.

[0157] Exemplarily, in some embodiments, the relevant syntax elements further include a second syntax element. The method further includes: setting the second syntax element based on whether the current image block allows the use of a second loop filter model based on a neural network; wherein the second loop filter model is a candidate loop filter model other than the first loop filter model; and when determining that the current image block where the current coding tree unit is located uses the second loop filter model based on a neural network, setting the first syntax element based on the target loop filter model of the current coding tree unit.

[0158] Exemplarily, the current image block includes at least one of the following: an image sequence where the current coding tree unit is located, an image where the current coding tree unit is located, a slice where the current coding tree unit is located, and a current image tree unit.

[0159] Exemplarily, the second syntax element includes at least one of the following: a second syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use the second loop filtering model based on the neural network; a second syntax element at the image level, used to indicate whether the image is allowed to use the second loop filtering model based on the neural network; a second syntax element at the slice level, used to indicate whether the slice is allowed to use the second loop filtering model based on the neural network; a second syntax element at the coding tree unit level, used to indicate whether the coding tree unit is allowed to use the second loop filtering model based on the neural network.

[0160] In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level. In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level and a second syntax element at a picture level. In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level and a second syntax element at a slice level. In some embodiments, the second syntax element comprises a second syntax element at a picture sequence level, a second syntax element at a picture level (or a slice level), and a second syntax element at a coding tree unit level.

[0161] Exemplarily, in some embodiments, the relevant syntax elements also include a third syntax element. The method also includes: setting the third syntax element according to whether the current image block where the current coding tree unit is located uses a neural network-based loop filtering technology; when determining that the current image block where the current coding tree unit is located uses a neural network-based loop filtering technology, setting the first syntax element according to the target loop filtering model of the current coding tree unit. In some embodiments, when the third syntax element is a first preset value, it is determined that all coding tree units in the current image block do not use a neural network-based loop filtering technology; when the third syntax element is a second preset value, it is determined that all coding tree units in the current image block use a neural network-based loop filtering technology; when the third syntax element is a second preset value, it is determined that some coding tree units in the current image block use a neural network-based loop filtering technology.

[0162] Exemplarily, in some embodiments, the third syntax element includes at least one of the following: a third syntax element at the image sequence level, used to indicate whether the image sequence uses a neural network-based loop filtering technology; a third syntax element at the image level, used to indicate whether the image uses a neural network-based loop filtering technology; a third syntax element at the slice level, used to indicate whether the slice uses a neural network-based loop filtering technology; a third syntax element at the coding tree unit level, used to indicate whether the coding tree unit uses a neural network-based loop filtering technology.

[0163] In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a picture level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a slice level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level, a third syntax element at a picture level (or a slice level), and a third syntax element at a coding tree unit level.

[0164] Exemplarily, in some embodiments, the relevant syntax elements also include a fourth syntax element. The method further includes: setting the fourth syntax element according to the image type or slice type of the current coding tree unit; when it is determined that the image type or slice type of the current coding tree unit is a preset type, setting the first syntax element according to the first syntax element and the target loop filter model of the current coding tree unit. That is, only when the slice type of the current coding tree unit is a preset type is it allowed to use the candidate loop filter model based on the neural network for loop filtering; otherwise, it is not allowed. In some embodiments, the preset type is B_slice.

[0165] In some embodiments, the fourth syntax element includes one of the following: a fourth syntax element at the picture sequence level, used to indicate the picture type or slice type of all coding tree units in the picture sequence; a fourth syntax element at the picture level, used to indicate the picture type or slice type of all coding tree units in the picture; a fourth syntax element at the slice level, used to indicate the picture type or slice type of all coding tree units in the slice; a fourth syntax element at the coding tree unit level, used to indicate the picture type or slice type of the coding tree unit.

[0166] In some embodiments, the relevant syntax elements further include a fifth syntax element. The method further includes: setting the fifth syntax element based on whether the current image block allows the use of a neural network-based loop filtering technique. Determining whether the neural network-based loop filtering technique is allowed, initializing the neural network-based loop filtering technique, loading a candidate neural network-based loop filtering model, calculating a distortion cost value, and determining a target loop filtering model; and determining, based on the fifth syntax element, whether the neural network-based loop filtering technique is not allowed, using another filtering technique, or not using any filtering technique.

[0167] In some embodiments, the fifth syntax element includes at least one of the following: a fifth syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the image level, used to indicate whether the image is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the slice level, used to indicate whether the slice is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the coding tree unit level, used to indicate whether the coding tree unit is allowed to use the neural network-based loop filtering technology.

[0168] In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a picture level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a slice level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level, a fifth syntax element at a picture level (or a slice level), and a fifth syntax element at a coding tree unit level.

[0169] In summary, when determining that the current image block uses a candidate loop filter model based on a neural network according to at least one of the second syntax element, the third syntax element, the fourth syntax element, and the fifth syntax element, the target loop filter model is determined from the candidate loop filter models according to the first syntax element.

[0170] In some embodiments, the parameter sample information further includes candidate quantization parameters; the method further includes determining, based on a first distortion cost value of the current coding tree unit, a target quantization parameter for the current image block in which the current coding tree unit resides from the candidate quantization parameters. In other words, the encoder can select multiple candidate quantization parameters and determine the target quantization parameter for the current coding tree unit by calculating a vector cost value. Depending on the adjustment parameters of the quantization parameters allowed for the current image or slice, or the input information for adjusting the allowed quantization parameters, the filtered reconstructed samples derived by the model inference may differ due to the different input information.

[0171] In some embodiments, the method further includes: determining whether to adjust the quantization parameter of the current image block where the current coding tree unit is located based on the target quantization parameter of the current image block where the current coding tree unit is located, and setting a sixth syntax element; determining to adjust the quantization parameter of the current image block where the current coding tree unit is located, and setting a seventh syntax element based on the target quantization parameter of the current image block where the current coding tree unit is located.

[0172] In some embodiments, the sixth syntax element includes at least one of the following: a sixth syntax element at the image sequence level, used to indicate whether to adjust the quantization parameters of the coding tree units in the image sequence; a sixth syntax element at the image level, used to indicate whether to adjust the quantization parameters of the coding tree units in the image; a sixth syntax element at the slice level, used to indicate whether to adjust the quantization parameters of the coding tree units in the slice; a sixth syntax element at the coding tree unit level, used to indicate whether to adjust the quantization parameters of the coding tree units.

[0173] In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a picture level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a slice level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level, a sixth syntax element at a picture level (or a slice level), and a sixth syntax element at a coding tree unit level.

[0174] The seventh syntax element includes one of the following: a seventh syntax element at the picture sequence level, used to indicate the adjusted quantization parameters of all coding tree units in the picture sequence; a seventh syntax element at the picture level, used to indicate the adjusted quantization parameters of all coding tree units in the picture; a seventh syntax element at the slice level, used to indicate the adjusted quantization parameters of all coding tree units in the slice; or a seventh syntax element at the coding tree unit level, used to indicate the adjusted quantization parameters of the coding tree unit.

[0175] In some embodiments, the reference sample information also includes at least one of the following: quantization parameters, boundary strength information of the current coding tree unit, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image, the slice type of the current coding tree unit and the partitioning information of the current coding tree unit.

[0176] In actual applications, the inputs of different loop filter models may not be exactly the same. The target loop filter model and the input information of the target loop filter model are determined based on relevant syntax elements. In some embodiments, the candidate loop filter models include a first loop filter model and a second loop filter model; wherein the reference sample information input to the first loop filter model also includes: quantization parameters and boundary strength information of the current coding tree unit; the reference sample information input to the second loop filter model also includes: quantization parameters, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the forward reference image, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the backward reference image.

[0177] Correspondingly, in some embodiments, the method further includes: when the candidate loop filtering model is the second loop filtering model, obtaining the first reference image of the first reference image list, and obtaining the first reference image of the second reference image list; when the first reference image of the first reference image list and the first reference image of the second reference image list are the same image, obtaining the second reference image of the first reference image list or the second reference image list; using the reference image of the first reference image list obtained as the forward reference image, and the reference image of the first reference image list as the backward reference image.

[0178] For the input of the second loop filter model with multiple reference image inputs, a duplicate check can be performed on the two reference images obtained from the two reference image lists to avoid inputting the same two coding tree unit information. If the two reference images are the same, the second candidate image from the first reference image list can be used as input; alternatively, the second candidate image from the second reference image list can be used as input.

[0179] In some embodiments, for a loop filter model with multiple reference images, the selection of the first reference image and the selection of the second reference image can be indicated by a flag in a syntax element. For example, if the first candidate image in the first reference image list is selected, the flag is position 1, and if the second candidate image is selected, the flag is position 2. The encoder performs rate-distortion cost optimization to determine which candidate image in the first reference image list to use. The same applies to the selection of the second reference image list.

[0180] In some embodiments, for a loop filtering model with multiple reference image inputs, if the candidate images in the reference image list are temporally distant from the current image, the codec can simultaneously select the reference image closest to the current image as input. Specifically, two reference image lists are traversed, and the candidate image closest to the current image is selected as the first reference image, and the next closest candidate image is selected as the second reference image.

[0181] Exemplarily, in some embodiments, the method further includes: when the temporal level of the current coding tree unit is greater than or equal to the temporal level threshold, determining the target loop filter model of the current coding tree unit to be the second loop filter model; and when the temporal level of the current coding tree unit is less than the temporal level threshold, determining the target loop filter model of the current coding tree unit from candidate loop filter models based on a neural network according to the relevant syntax elements. Exemplarily, when the encoder encodes a picture or slice of a high temporal level, the encoder uses the Alter_B_Slice model by default, and therefore does not need to write slice_Alter_B_flag into the bitstream. When the decoder decodes a picture or slice of a high temporal level, it does not need to parse the bitstream to obtain information about slice_Alter_B_flag, and by default slice_Alter_B_flag is set to true and the Alter_B_Slice model is used.

[0182] For example, in some embodiments, the current coding tree unit is a maximum coding unit, or is obtained by rescaling the maximum coding unit. That is, patchWidth and patchHeight are the block sizes input to the model, which can be the coding tree unit size or a larger patch block after padding the coding tree unit.

[0183] This paper proposes a neural network-based loop filtering model selection scheme, that is, when performing neural network loop filtering on a B_Slice image or slice, the encoder selects one of the Alter_B_Slice model and the B_Slice model to be applied to the current image or slice, and writes a new syntax element indicating which of the Alter_B_Slice model or the B_Slice model is used into the bitstream and transmits it to the decoder. The decoder parses the bitstream and obtains the relevant syntax element, that is, whether the current image or slice should use the Alter_B_Slice model or the B_Slice model. The above-mentioned information on using the Alter_B_Slice model or the B_Slice model can be expressed at the image level or slice level through a new syntax element, such as slice_Alter_B_flag. For the sake of convenience and writing, this syntax element is recorded as slice_Alter_B_flag here. Its semantics can be expressed as follows: if this flag is zero or false, it indicates that the current picture or slice does not use the Alter_B_Slice model and uses the original B_Slice model; if this flag is non-zero or true, it indicates that the current picture or slice uses the Alter_B_Slice model. If the slice_Alter_B_flag syntax element is not present in the bitstream, the variable corresponding to this syntax element defaults to zero or false.

[0184] The encoder selects one of the Alter_B_Slice model and the original B_Slice model as the current image or slice by first filtering the current image using the original B_Slice model and calculating its distortion cost 1. Then, it filters the current image using the Alter_B_Slice model and calculating its distortion cost 2. By comparing the two cost values, the more optimal model is selected as the neural network filtering model for the current image or slice. It is worth noting that different encoder methods are possible; this article only lists these methods for ease of understanding and presentation.

[0185] The newly added syntax element for selecting the Alter_B_Slice model or the original B_Slice model can be an image-level or slice-level syntax element, and its meaning is to determine the neural network loop filter model used by the current image or slice by the variable value corresponding to the syntax element. However, it is worth noting that the newly added syntax element can also be a syntax element at the coding tree unit level, and its meaning is to determine the neural network loop filter model used by the current coding tree unit by the variable value corresponding to the syntax element.

[0186] In this embodiment, different quantization parameters can be selected at the coding tree unit level, and the quantization parameters are usually used as constant inputs of the model. This embodiment focuses on combining the adjustment of the quantization parameters with the selection of the model.

[0187] Furthermore, to improve the performance and complexity of this model, this paper proposes a method for selecting models based on the temporal hierarchy for B_Slice images or slices in non-low-latency test conditions. For B_Slices in low-latency test conditions, the selection is made at the encoder end. Specifically, for high-temporal hierarchy codecs, the Alter_B_Slice model is used by default, without requiring the codec to parse newly added syntax elements to obtain model usage information. However, for low-temporal hierarchy codecs, the codec needs to parse newly added syntax elements in the bitstream to obtain model usage information.

[0188] This paper also proposes a method for improving coding performance by performing a duplicate check on the reference image information obtained from the first reference image list and the reference image information obtained from the second reference image list. If the two reference images are identical, that is, the second reference image in the second reference image list is used as the backward reference image input information for the model. Alternatively, if the two reference images are identical, the second reference image from the first reference image list can be used as the forward reference image input information for the model. This method is not further described here.

[0189] The various neural network-based loop filtering schemes proposed in this paper will introduce one or more new syntax elements for selecting the loop filtering model. These syntax elements include but are not limited to the following:

[0190] The second syntax element of the picture sequence level sps_nnlf_alter_enable_flag (sequence level switch flag)

[0191] The first syntax element slice_Alter_B_flag at the picture level or slice level (picture level or slice level flag)

[0192] The sixth syntax element slice_nnlf_param_flag at the picture level or slice level (picture level or slice level flag)

[0193] The seventh syntax element ctb_nnlf_param at the coding tree unit level (coding tree unit level flag)

[0194] The following further illustrates an embodiment of the encoding method.

[0195] The encoder predicts the current image and obtains a prediction block for each coding unit. The residual of the coding unit is obtained by subtracting the original image block from the prediction block. The residual is transformed using various transform modes to obtain frequency-domain residual coefficients. This residual is then quantized and inversely quantized, and inversely transformed to obtain distorted residual information. This distorted residual information is superimposed on the prediction block to obtain a reconstructed block. The term "reconstructed block" here generally refers to a coding unit consisting of reconstructed samples obtained by superimposing the predicted samples with the inversely transformed and inversely quantized residual samples. Subsequently, the loop filter module filters the image at the coding tree unit level, where the proposed technology is applied.

[0196] The coding tree unit enters the neural network loop filter module and obtains the neural network-based loop filter enable flag, sps_nnlf_enable_flag. If this flag is true, the neural network-based loop filter technology is enabled; if it is false, the neural network-based loop filter technology is not enabled. The sequence-level enable flag needs to be written into the bitstream when encoding the video sequence. Here, the technology in this paper can add a new sequence-level enable flag, such as the new syntax element sps_nnlf_alter_enable_flag. The naming of this syntax element is mainly for ease of understanding and textual content. It can be modified in actual applications and standard texts, but its semantic content should be consistent or similar. Both indicate that if this flag is true, the neural network-based loop filter model selection scheme proposed in this paper is enabled; if this flag is false, the neural network-based loop filter model selection scheme proposed in this paper is not enabled.

[0197] 1. If the flag bit for allowing the use of neural network-based loop filtering is true, the encoder attempts the neural network-based loop filtering technology, that is, executes 2. If the flag bit for allowing the use of neural network-based loop filtering is false, the encoder does not attempt the neural network-based loop filtering technology, that is, skips 2 and directly executes 3.

[0198] 2. Initialize the neural network-based loop filtering technology and load the neural network model suitable for the current image.

[0199] a. Calculate the cost of the reconstruction sample

[0200] The encoder calculates the cost information without using the neural network loop filtering technology. That is, it uses the reconstructed samples of the coding tree unit to be used as network input and the original image samples of the coding tree unit to calculate and record the rate-distortion cost value. The rate-distortion cost value mainly consists of the absolute error and the number of bits consumed. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice and accumulates and records it to obtain costRec. The costRec cost information represents the accumulated cost of all coding tree units in the current image or slice.

[0201] b. Filter and calculate the cost of the two models

[0202] The encoder attempts to use a neural network-based loop filtering technique, using the reconstructed sample rec, predicted sample pred, quantization parameter QP, and boundary strength information BS of the current coding tree unit to input into the loaded original B_Slice model for inference. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost of the filtered reconstructed sample of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice and accumulates and records it to obtain costOrgM. Similarly, costOrgM also represents the accumulated cost of all coding tree units in the current image or slice.

[0203] If the current image or slice type is B_Slice, continue to execute the remaining steps; otherwise, skip the remaining steps, assign costOrgM to costCnn and execute 2.c;

[0204] The encoding end continues to try the neural network-based loop filtering technology, using the reconstructed sample rec of the current coding tree unit, the predicted sample pred, the quantization parameter QP, the reconstructed sample Forw of the coding tree unit at the corresponding position in the first reference image, and the reconstructed sample Bacw of the coding tree unit at the corresponding position in the second reference image to input into the loaded Alter_B_Slice model for inference. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost value of the reconstructed sample after filtering of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all the coding tree units in the current image or slice and accumulates and records them to obtain costAlterM. Similarly, costAlterM also represents the accumulated cost of all the coding tree units in the current image or slice.

[0205] If costAlterM is less than costOrgM, it means that the filtering effect of the current image or slice using the Alter_B_Slice model is better than that of the original B_Slice model, and the newly added flag slice_Alter_B_flag is set to true; otherwise, it means that the filtering effect of the current image using the original B_Slice model is better than that of the Alter_B_Slice model, and the newly added flag slice_Alter_B_flag is set to false; the smaller of costAlterM and costOrgM is assigned to costCnn to facilitate subsequent understanding and writing.

[0206] c. Final information of decision neural network loop filter module

[0207] The encoder is experimenting with optimizing the selection at the coding tree unit level. In the second round of attempts at neural network loop filtering (NNLF), the encoder defaulted to using NNLF for all coding tree units in the current image. This was controlled by a single image-level flag, slice_nnlf_flag, while the ctb_nnlf_flag was not required at the coding tree unit level. Now, an attempt is being made to switch the flag at the coding tree unit level, with each coding tree unit having its own unique flag.

[0208] The encoder traverses the coding tree units, compares the cost information of each coding tree unit before filtering with the neural network loop filtering technology and the cost information after filtering. If the unfiltered cost information is smaller, the usage flag of the coding tree unit is set to false; otherwise, the usage flag of the coding tree unit is set to true. After the encoder has traversed all the coding tree units, it accumulates the minimum cost information of each coding tree unit to obtain the rate-distortion cost value costCtu of the current image or slice. The costCtu represents the accumulated value of the minimum cost of all coding tree units in the current image or slice, and the model used by the neural network loop filtering technology here is the better model determined in 2.b. By reading the variable information of the slice_Alter_B_flag flag, it can be obtained that the Alter_B_Slice model or the original B_Slice model is more suitable for the current image or slice.

[0209] If costRec is the minimum, it means that the current image or slice is better not to use the neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 0 and written into the bitstream for transmission to the decoder, indicating that all coding tree units in the current image or slice do not use the neural network loop filtering technology, and the coding tree unit-level flag does not need to be written into the bitstream;

[0210] If costCnn is the minimum, it means that all coding tree units in the current image or slice use neural network loop filtering technology, and the image-level or slice-level flag slice_nnlf_flag is set to 1 and written into the bitstream for transmission to the decoder, and the coding unit-level flag does not need to be written into the bitstream; In addition, if the current image or slice type is B_Slice, the information of slice_Alter_B_flag needs to be written into the bitstream for transmission to the decoder to indicate that the current image or slice needs to use the Alter_B_Slice model or the original B_Slice model;

[0211] If costCtu is the minimum, it means that not all coding tree units in the current image or slice use the neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 2 and written into the bitstream for transmission to the decoding end, and the neural network loop filtering technology of each coding tree unit needs to be written into the bitstream using the flag ctb_nnlf_flag; in addition, if the current image or slice type is B_Slice, the slice_Alter_B_flag information needs to be written into the bitstream for transmission to the decoding end to indicate that the current image or slice needs to use the Alter_B_Slice model or the original B_Slice model.

[0212] 3. The encoder continues to try other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article and will not be explained in detail here.

[0213] The encoder predicts the current image and obtains a prediction block for each coding unit. The residual of the coding unit is obtained by subtracting the original image block from the prediction block. The residual is transformed using various transform modes to obtain frequency-domain residual coefficients. This residual is then quantized and inversely quantized, and inversely transformed to obtain distorted residual information. This distorted residual information is superimposed on the prediction block to obtain a reconstructed block. The term "reconstructed block" here generally refers to a coding unit consisting of reconstructed samples obtained by superimposing the predicted samples with the inversely transformed and inversely quantized residual samples. Subsequently, the loop filter module filters the image at the coding tree unit level, where the proposed technology is applied.

[0214] The coding tree unit enters the neural network loop filter module and obtains the neural network-based loop filter enablement flag, sps_nnlf_enable_flag. If this flag is true, the neural network-based loop filter technology is enabled; if it is false, the neural network-based loop filter technology is not enabled. The sequence-level enablement flag needs to be written into the bitstream when encoding the video sequence. Here, the technology in this paper can add a new sequence-level enablement flag, such as the new syntax element sps_nnlf_alter_enable_flag. The naming of this syntax element is mainly for ease of understanding and textual content. It can be modified in actual applications and standard texts, but its semantic content should be consistent or similar. Both indicate that if this flag is true, the neural network-based loop filter model selection scheme proposed in this paper is enabled; if this flag is false, the neural network-based loop filter model selection scheme proposed in this paper is not enabled.

[0215] 1. If the flag bit for allowing the use of neural network-based loop filtering is true, the encoder attempts the neural network-based loop filtering technology, that is, executes 2. If the flag bit for allowing the use of neural network-based loop filtering is false, the encoder does not attempt the neural network-based loop filtering technology, that is, skips 2 and directly executes 3.

[0216] 2. Initialize the neural network-based loop filtering technology and load the neural network model suitable for the current image.

[0217] a. Calculate the cost of the reconstruction sample

[0218] The encoder calculates the cost information without using the neural network loop filtering technology. That is, it uses the reconstructed samples of the coding tree unit to be used as network input and the original image samples of the coding tree unit to calculate and record the rate-distortion cost value. The rate-distortion cost value mainly consists of the absolute error and the number of bits consumed. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice and accumulates and records it to obtain costRec. The costRec cost information represents the accumulated cost of all coding tree units in the current image or slice.

[0219] b. Filter and calculate the cost of the two models

[0220] The encoder attempts to use a neural network-based loop filtering technique, using the reconstructed sample rec, predicted sample pred, quantization parameter QP, and boundary strength information BS of the current coding tree unit to input into the loaded original B_Slice model for inference. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost of the filtered reconstructed sample of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice and accumulates and records it to obtain costOrgM. Similarly, costOrgM also represents the accumulated cost of all coding tree units in the current image or slice.

[0221] If the current image or slice type is B_Slice, continue to execute the remaining steps; otherwise, skip the remaining steps, assign costOrgM to costCnn and execute 2.c;

[0222] The encoding end continues to try the neural network-based loop filtering technology, using the reconstructed sample rec of the current coding tree unit, the predicted sample pred, the quantization parameter QP, the reconstructed sample Forw of the coding tree unit at the corresponding position in the first reference image, and the reconstructed sample Bacw of the coding tree unit at the corresponding position in the second reference image to input into the loaded Alter_B_Slice model for inference. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost value of the reconstructed sample after filtering of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all the coding tree units in the current image or slice and accumulates and records them to obtain costAlterM. Similarly, costAlterM also represents the accumulated cost of all the coding tree units in the current image or slice.

[0223] If costAlterM is less than costOrgM, it means that the filtering effect of the current image or slice using the Alter_B_Slice model is better than that of the original B_Slice model, and the newly added flag slice_Alter_B_flag is set to true; otherwise, it means that the filtering effect of the current image using the original B_Slice model is better than that of the Alter_B_Slice model, and the newly added flag slice_Alter_B_flag is set to false, and the smaller of costAlterM and costOrgM is assigned to costCnnQP0 to facilitate subsequent understanding and text;

[0224] The above steps are the same as those in Example 1, but in Example 2, adjustments can be made to the input part, such as adjusting the quantization parameter QP. The input information of the model is modified according to the QP adjustment parameters allowed by the current image or slice or the QP parameters allowed to be used. Due to the difference in input information, the model inference and the filtered reconstructed samples obtained are also different. In this embodiment, the unadjusted quantization parameter information is recorded as QP0, and the candidate adjustment quantization parameter information is recorded as QP1 and QP2. The determined models are used for QP1 and QP2 respectively to obtain the rate-distortion costs costCnnQP1 and costCnnQP2 of the current image or slice. The steps are the same as above, and the quantization parameter information is modified only on the input side.

[0225] c. Final information of decision neural network loop filter module

[0226] The encoder is experimenting with optimizing the selection at the coding tree unit level. In the second round of attempts at neural network loop filtering (NNLF), the encoder defaulted to using NNLF for all coding tree units in the current image. This was controlled by a single image-level flag, slice_nnlf_flag, while the ctb_nnlf_flag was not required at the coding tree unit level. Now, an attempt is being made to switch the flag at the coding tree unit level, with each coding tree unit having its own unique flag.

[0227] The encoder traverses the coding tree units (CTUs) and compares the cost information of each CTU before filtering using the neural network loop filtering technique with the cost information after filtering. At this point, there are three types of reconstructed sample blocks after filtering: QP0 (with the unmodified input quantization parameter), and QP1 and QP2 (with the modified input quantization parameters). When comparing the reconstructed sample costs of various CTUs, all must be considered. If the unfiltered cost information is smaller, the usage flag of the CTU is set to false; otherwise, the usage flag of the CTU is set to true. If the reconstructed sample cost of the CTU corresponding to QP0 is the smallest, the parameter flag ctb_nnlf_param for the current CTU is set to 0. If the reconstructed sample cost of the CTU corresponding to QP1 is the smallest, the parameter flag ctb_nnlf_param for the current CTU is set to 1. Otherwise, if the reconstructed sample cost of the CTU corresponding to QP2 is the smallest, the parameter flag ctb_nnlf_param for the current CTU is set to 2.

[0228] After traversing all coding tree units (CTUs), the encoder accumulates the minimum cost information for each CTU to obtain the rate-distortion cost (costCtu) for the current image or slice. This costCtu represents the cumulative value of the minimum cost of all CTUs in the current image or slice. The neural network loop filter technology uses the optimal model determined in 2.b. By reading the slice_Alter_B_flag flag, the Alter_B_Slice model or the original B_Slice model can be determined to be more appropriate for the current image or slice.

[0229] If costRec is the minimum, it means that the current image or slice is better not to use the neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 0 and written into the bitstream for transmission to the decoder, indicating that all coding tree units in the current image or slice do not use the neural network loop filtering technology, and the coding tree unit-level flag does not need to be written into the bitstream;

[0230] If costCnnQP0 / QP1 / QP2 is the minimum, it means that all coding tree units in the current image or slice use neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 1 and written into the bitstream for transmission to the decoder. The coding unit-level flag does not need to be written into the bitstream. If costCnnQP0 is the minimum, the image-level or quantization parameter-level flag slice_nnlf_param is set to 0 and written into the bitstream for transmission to the decoder. If costCnnQP1 is the minimum, the image-level or quantization parameter-level flag slice_nnlf_param is set to 1 and written into the bitstream for transmission to the decoder. Otherwise, that is, if costCnnQP2 is the minimum, the image-level or quantization parameter-level flag slice_nnlf_param is set to 2 and written into the bitstream for transmission to the decoder. In addition, if the current image or slice type is B_Slice, the slice_Alter_B_flag information needs to be written into the code stream and transmitted to the decoder to indicate that the current image or slice needs to use the Alter_B_Slice model or the original B_Slice model;

[0231] If costCtu is the minimum, it means that not all coding tree units in the current image or slice use neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 2 and written into the bitstream for transmission to the decoder. The neural network loop filtering technology usage flag ctb_nnlf_flag of each coding tree unit needs to be written into the bitstream; and the quantization parameter modification information index ctb_nnlf_param of each coding tree unit needs to be written into the bitstream. In addition, if the current image or slice type is B_Slice, the slice_Alter_B_flag information needs to be written into the bitstream for transmission to the decoder to indicate that the current image or slice needs to use the Alter_B_Slice model or the original B_Slice model.

[0232] 3. The encoder continues to try other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article and will not be explained in detail here.

[0233] The embodiment of the present application may also use residual scaling technology. The residual scaling technology is applied to the output of the neural network model, and the residual obtained by subtracting the filtered reconstructed samples output by the neural network model from the unfiltered reconstructed samples is scaled.

[0234] In all the above embodiments, it is not specified whether it is a luminance component or a chrominance component, because the method proposed in this article is applicable to both luminance components and chrominance components, so there is no limitation here. The syntax elements in the article can be clearly known, such as ctb_nnlf_flag and slice_nnlf_flag, which can be divided into separate control of luminance components and chrominance components. It is worth noting that, for example, ctb_nnlf_param can be different from the separate control of each color component mentioned above. Here, in order to save bit overhead, the same flag bit can be used between color components. Specifically, ctb_nnlf_param can simultaneously indicate that the luminance component and the chrominance component make the same adjustment, which is not elaborated in detail here.

[0235] In all of the above embodiments, the parsing conditions for the picture-level or slice-level flag slice_Alter_B_flag may differ from those in the above embodiments. Specifically, when the encoder encodes a picture or slice at a higher temporal level, the encoder uses the Alter_B_Slice model by default, and therefore does not need to write slice_Alter_B_flag into the bitstream. When the decoder decodes a picture or slice at a higher temporal level, it does not need to parse the bitstream to obtain information about slice_Alter_B_flag. Slice_Alter_B_flag is set to true by default and the Alter_B_Slice model is used.

[0236] In all the above embodiments, for the input part of the Alter_B_Slice model, the first reference image can be the first candidate image in the first reference image list, or the second candidate image in the first reference image list, etc.; similarly, the second reference image can be the first candidate image in the second reference image list, or the second candidate image in the second reference image list.

[0237] In all the above embodiments, for the input portion of the Alter_B_Slice model, a duplicate check operation can be performed on the first and second reference images input to avoid inputting two identical coding tree unit information. If the two reference images are the same, the second candidate image in the second reference image list can be obtained as the input of the second reference image; the third candidate image in the second reference image list can also be obtained as the input of the second reference image; similarly, the second candidate image in the first reference image list can also be used as the input of the first reference image. This article will not elaborate on this.

[0238] In all of the above embodiments, for the input portion of the Alter_B_Slice model, the first reference image input and / or the second reference image input can be specified as an I_Slice full intra-frame coded frame. In the inter-frame coded portion, the I_Slice is often encoded with higher quality parameters so that the subsequent B_Slice can serve as a reference to improve overall performance. Therefore, the I_Slice often contains more high-frequency information than the B_Slice, so the input portion of the Alter_B_Slice model can be specified here as the I_Slice reference image. This is not limited here.

[0239] In yet another embodiment of the present application, the embodiment of the present application further provides a code stream, which is generated by bit encoding according to information to be encoded; wherein the information to be encoded includes at least: relevant syntax elements of the current coding tree unit.

[0240] The encoder uses a candidate loop filter module to filter the coding tree unit, calculates the distortion cost, determines the target loop filter model corresponding to the minimum distortion cost, and writes the relevant syntax elements into the bitstream. The decoder only needs to parse the bitstream and select the optimal target loop filter model for the current coding tree unit based on the relevant syntax elements, thereby improving the filtering performance of the reconstructed samples of the current coding tree unit.

[0241] Table 3

[0242]

[0243] The experimental results show that the proposed method has a performance improvement of nearly 1% for the luminance component under the test conditions of Low Delay B, and has no decoding time complexity.

[0244] In yet another embodiment of the present application, referring to FIG10 , a schematic diagram of the structure of a decoding device provided in an embodiment of the present application is shown. As shown in FIG10 , the decoding device may include:

[0245] The decoding unit 1001 is configured to decode the code stream and determine the relevant syntax elements of the current coding tree unit;

[0246] A first determining unit 1002 is configured to determine a target loop filter model of a current coding tree unit from candidate loop filter models based on a neural network according to the relevant syntax elements;

[0247] The second determining unit 1003 is configured to determine reference sample information of the current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0248] The filtering unit 1004 is configured to input the reference sample information of the current coding tree unit into the first loop filtering model for filtering, and output the filtered reconstructed sample information.

[0249] In practical applications, an embodiment of the present application further provides a decoding device. FIG11 is a schematic diagram of a specific hardware structure of a decoding device provided in an embodiment of the present application. As shown in FIG11 , the decoding device includes:

[0250] A first memory 1101 and a first processor 1102; the first memory 1101 stores a computer program that can be run on the first processor 1102, and the first processor 1102 executes the program in accordance with the encoding method on the encoder side.

[0251] In practical applications, the decoding device may further include: a first communication interface, used for sending and receiving signals during the process of sending and receiving information with other external network elements.

[0252] In another embodiment of the present application, referring to FIG12 , a schematic diagram of the composition structure of an encoding device provided in an embodiment of the present application is shown. As shown in FIG12 , the encoding device may include:

[0253] The first determining unit 1201 is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0254] The filtering unit 1202 is configured to input the reference sample information of the current coding tree unit into the candidate loop filter model based on the neural network for filtering, and output the filtered reconstructed sample information;

[0255] The second determining unit 1203 is configured to determine a first distortion cost value of the current coding tree unit based on the original sample information of the current coding tree unit and the filtered reconstructed sample information; and determine a target loop filter model for the current coding tree unit from candidate loop filter models based on a neural network based on the first distortion cost value of the current coding tree unit;

[0256] The encoding unit 1204 is configured to encode syntax elements related to the target loop filter model of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0257] In practical applications, an embodiment of the present application further provides an encoding device. FIG13 is a schematic diagram of a specific hardware structure of an encoding device provided in an embodiment of the present application. As shown in FIG13 , the encoding device includes:

[0258] The second memory 1301 and the second processor 1302; the second memory 1301 stores a computer program that can be run on the second processor 1302, and the second processor 1302 executes the program when the decoder side decodes the method.

[0259] In practical applications, the encoding device may further include: a second communication interface for receiving and sending signals during the process of sending and receiving information with other external network elements.

[0260] Using the above-mentioned device or apparatus, the encoder uses a candidate loop filter module to filter the coding tree unit, calculates the distortion cost value, determines the target loop filter model corresponding to the minimum distortion cost value, and writes the relevant syntax elements into the bitstream. The decoder only needs to parse the bitstream and select the optimal target loop filter model for the current coding tree unit based on the relevant syntax elements, thereby improving the filtering performance of the reconstructed samples of the current coding tree unit.

[0261] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0262] In yet another embodiment of the present application, see Figure 14, which shows a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. As shown in Figure 14, the coding and decoding system 140 may include an encoder 1401 and a decoder 1402. The encoder 1401 may be a device integrated with the encoding device described in the previous embodiment, or may also be the encoding device described in the previous embodiment; the decoder 1402 may be a device integrated with the decoding device described in the previous embodiment, or may also be the decoding device described in the previous embodiment.

[0263] In the embodiment of the present application, in the codec system 140, the encoder uses a candidate loop filter module to filter the coding tree unit, calculates the distortion cost value, determines the target loop filter model corresponding to the minimum distortion cost value, and writes the relevant syntax elements into the bitstream. The decoder only needs to parse the bitstream and select the optimal target loop filter model for the current coding tree unit based on the relevant syntax elements, thereby improving the filtering performance of the reconstructed samples of the current coding tree unit.

[0264] Accordingly, an embodiment of the present invention further provides a computer storage medium storing a computer program, which, when executed by a first processor, implements a decoding method of a decoder; or, when executed by a second processor, implements an encoding method of an encoder.

[0265] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding. It should be noted that "first", "second", etc. are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0266] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments. The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. Industrial Applicability

[0267] The present application provides a coding and decoding method, apparatus, coding device, decoding device, and storage medium, wherein the decoding method includes: decoding a bitstream, determining relevant syntax elements of a current coding tree unit; determining a target loop filter model of the current coding tree unit from candidate loop filter models based on a neural network based on the relevant syntax elements; determining reference sample information of the current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit; inputting the reference sample information of the current coding tree unit into the target loop filter model for filtering, and outputting filtered reconstructed sample information. The encoding end uses a candidate loop filter module to filter the coding tree unit, calculates a distortion cost value, determines a target loop filter model corresponding to the minimum distortion cost value, and writes the relevant syntax elements into the bitstream. The decoding end only needs to parse the bitstream and select the optimal target loop filter model for the current coding tree unit based on the relevant syntax elements, thereby improving the filtering performance of the reconstructed samples of the current coding tree unit.

Claims

1. A decoding method, the method comprising: Decode the code stream and determine the relevant syntax elements of the current coding tree unit; Determining a target loop filter model of a current coding tree unit from candidate loop filter models based on a neural network according to the relevant syntax elements; Determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit; The reference sample information of the current coding tree unit is input into the target loop filter model for filtering, and the filtered reconstructed sample information is output.

2. The method according to claim 1, wherein: include: According to the first syntax element, a target loop filter model of a current coding tree unit is determined from candidate loop filter models based on a neural network.

3. The method according to claim 2, wherein: include: The first syntax element includes one of the following: The first syntax element at the picture sequence level is used to indicate the target loop filter model for all coding tree units in the picture sequence; The first syntax element at the picture level is used to indicate the target loop filter model for all coding tree units in the picture; The first syntax element at the slice level is used to indicate the target loop filter model for all coding tree units in the slice; The first syntax element at the coding tree unit level is used to indicate the target loop filter model of the coding tree unit.

4. The method according to claim 2, wherein: include: Determine, according to the second syntax element, whether the current image block where the current coding tree unit is located is allowed to use a second loop filter model based on a neural network; wherein the second loop filter model is a candidate loop filter model; When it is determined according to the second syntax element that the current image block where the current coding tree unit is located uses a second loop filtering model based on a neural network, a target loop filtering model of the current coding tree unit is determined from candidate loop filtering models based on a neural network according to the first syntax element.

5. The method according to claim 4, wherein: The second syntax element includes at least one of the following: A second syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use a second loop filter model based on a neural network; A second syntax element at the image level, used to indicate whether the image allows the use of a second loop filter model based on a neural network; A second syntax element at the slice level, used to indicate whether the slice is allowed to use a second loop filter model based on a neural network; The second syntax element at the coding tree unit level is used to indicate whether the coding tree unit is allowed to use the second loop filter model based on the neural network.

6. The method according to any one of claims 2 to 5, wherein: Also includes: Determining, according to the third syntax element, whether a current image block where the current coding tree unit is located uses a loop filtering technology based on a neural network; When it is determined according to the third syntax element that the current image block where the current coding tree unit is located uses a neural network-based loop filtering technology, a target loop filtering model of the current coding tree unit is determined from candidate loop filtering models based on the neural network according to the first syntax element.

7. The method according to claim 6, wherein: The third syntax element includes at least one of the following: The third syntax element at the image sequence level is used to indicate whether the image sequence uses a neural network-based loop filtering technique; The third syntax element at the image level is used to indicate whether the image uses a neural network-based loop filtering technique; The third syntax element at the slice level is used to indicate whether the slice uses a neural network-based loop filtering technique; The third syntax element at the coding tree unit level is used to indicate whether the coding tree unit uses a neural network-based loop filtering technique.

8. The method according to any one of claims 2 to 7, wherein: Also includes: Determine, according to the fourth syntax element, a picture type or a slice type of a current coding tree unit; When it is determined according to the fourth syntax element that the image type or slice type of the current coding tree unit is a preset type, a target loop filter model of the current coding tree unit is determined from candidate loop filter models based on a neural network according to the first syntax element.

9. The method according to any one of claims 2 to 8, wherein: include: According to the fifth syntax element, it is determined whether the current image block allows the use of a neural network-based loop filtering technique.

10. The method according to any one of claims 1 to 9, wherein: The parameter sample information also includes: a quantization parameter; the method also includes: Determining whether to adjust a quantization parameter of a current image block where a current coding tree unit is located according to a sixth syntax element; When determining, according to the sixth syntax element, to adjust the quantization parameter of the current image block where the current coding tree unit is located, determining, according to the seventh syntax element, the adjusted quantization parameter of the current image block where the current coding tree unit is located; The adjusted quantization parameter is input into the target loop filter model.

11. The method according to claim 10, wherein: The sixth syntax element includes at least one of the following: The sixth syntax element at the picture sequence level is used to indicate whether to adjust the quantization parameters of the coding tree units in the picture sequence; The sixth syntax element at the picture level is used to indicate whether to adjust the quantization parameters of the coding tree units in the picture; The sixth syntax element at the slice level is used to indicate whether to adjust the quantization parameters of the coding tree units in the slice; The sixth syntax element at the coding tree unit level is used to indicate whether to adjust the quantization parameter of the coding tree unit; The seventh syntax element includes one of the following: The seventh syntax element at the picture sequence level is used to indicate the adjusted quantization parameters of all coding tree units in the picture sequence; The seventh syntax element at the picture level is used to indicate the adjusted quantization parameters of all coding tree units in the picture; The seventh syntax element at the slice level is used to indicate the adjusted quantization parameters of all coding tree units in the slice; The seventh syntax element at the coding tree unit level is used to indicate the adjusted quantization parameter of the coding tree unit.

12. The method according to any one of claims 4 to 11, wherein: The current image block includes at least one of the following: an image sequence where the current coding tree unit is located, an image where the current coding tree unit is located, a slice where the current coding tree unit is located, and a current image tree unit.

13. The method according to claim 1, wherein: The reference sample information also includes at least one of the following: a quantization parameter, boundary strength information of the current coding tree unit, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image, a slice type of the current coding tree unit, and partition information of the current coding tree unit.

14. The method according to claim 13, wherein: The candidate loop filter models include a first loop filter model and a second loop filter model; wherein, The reference sample information input by the first loop filter model also includes: quantization parameter and boundary strength information of the current coding tree unit; The reference sample information input by the second loop filtering model also includes: quantization parameter, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the forward reference image, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the backward reference image.

15. The method according to claim 14, wherein: include: When the target loop filter model is the second loop filter model, obtaining a first reference image in the first reference image list, and obtaining a first reference image in the second reference image list; When the first reference image in the first reference image list and the first reference image in the second reference image list are the same image, acquiring the second reference image in the first reference image list or the second reference image list; The reference image in the first reference image list is used as the forward reference image, and the reference image in the first reference image list is used as the backward reference image.

16. The method according to any one of claims 1 to 15, wherein: include: When the time domain level of the current coding tree unit is greater than or equal to the time domain level threshold, determining that the target loop filter model of the current coding tree unit is a second loop filter model; When the time domain level of the current coding tree unit is less than the time domain level threshold, a target loop filter model of the current coding tree unit is determined from candidate loop filter models based on a neural network according to the relevant syntax elements.

17. The method according to any one of claims 1 to 16, wherein: The current coding tree unit is a maximum coding unit, or is obtained by scaling the maximum coding unit.

18. A coding method, the method comprising: Determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit; Input the reference sample information of the current coding tree unit into the candidate loop filter model based on the neural network for filtering, and output the reconstructed sample information after filtering; Determine a first distortion cost value of the current coding tree unit according to the original sample information of the current coding tree unit and the filtered reconstructed sample information; Determining a target loop filter model of the current coding tree unit from candidate loop filter models based on a neural network according to a first distortion cost value of the current coding tree unit; The relevant syntax elements of the target loop filter model of the current coding tree unit are encoded, and the obtained coded bits are written into the bitstream.

19. The method according to claim 18, wherein: include: A first syntax element is set according to a target loop filter model of a current coding tree unit.

20. The method according to claim 19, wherein: include: The first syntax element includes one of the following: The first syntax element at the picture sequence level is used to indicate the target loop filter model for all coding tree units in the picture sequence; The first syntax element at the picture level is used to indicate the target loop filter model for all coding tree units in the picture; The first syntax element at the slice level is used to indicate the target loop filter model for all coding tree units in the slice; The first syntax element at the coding tree unit level is used to indicate the target loop filter model of the coding tree unit.

21. The method according to claim 19, wherein: include: Setting a second syntax element according to whether the current image block allows the use of a second loop filter model based on a neural network; wherein the second loop filter model is a candidate loop filter model other than the first loop filter model; When it is determined that the current image block where the current coding tree unit is located uses a second loop filter model based on a neural network, a first syntax element is set according to a target loop filter model of the current coding tree unit.

22. The method according to claim 21, wherein: The second syntax element includes at least one of the following: A second syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use a second loop filter model based on a neural network; A second syntax element at the image level, used to indicate whether the image allows the use of a second loop filter model based on a neural network; A second syntax element at the slice level, used to indicate whether the slice is allowed to use a second loop filter model based on a neural network; The second syntax element at the coding tree unit level is used to indicate whether the coding tree unit is allowed to use the second loop filter model based on the neural network.

23. The method according to any one of claims 19 to 22, wherein: Also includes: Setting a third syntax element according to whether a current image block where the current coding tree unit is located uses a loop filtering technology based on a neural network; When it is determined that the current image block where the current coding tree unit is located uses a loop filtering technology based on a neural network, a first syntax element is set according to a target loop filtering model of the current coding tree unit.

24. The method according to claim 23, wherein: The third syntax element includes at least one of the following: The third syntax element at the image sequence level is used to indicate whether the image sequence uses a neural network-based loop filtering technique; The third syntax element at the image level is used to indicate whether the image uses a neural network-based loop filtering technique; The third syntax element at the slice level is used to indicate whether the slice uses a neural network-based loop filtering technique; The third syntax element at the coding tree unit level is used to indicate whether the coding tree unit uses a neural network-based loop filtering technique.

25. The method according to any one of claims 19 to 24, wherein: Also includes: Setting a fourth syntax element according to a picture type or a slice type of a current coding tree unit; When it is determined that the image type or the slice type of the current coding tree unit is a preset type, the first syntax element is set according to the first syntax element and according to the target loop filter model of the current coding tree unit.

26. The method according to any one of claims 19 to 25, wherein: include: The fifth syntax element is set according to whether the current image block allows the use of a neural network-based loop filtering technique.

27. The method according to any one of claims 18 to 26, wherein: The step of determining a target loop filter model of the current coding tree unit from candidate loop filter models based on a neural network according to a first distortion cost value of the current coding tree unit comprises: Accumulating the first distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the first distortion cost value of the current image block; According to the first distortion cost value of the current image block corresponding to the candidate loop filter model, a target loop filter model of the current coding tree unit is determined.

28. The method according to claim 27, wherein: Also includes: Determine a second distortion cost value of the current coding tree unit according to original sample information of the current coding tree unit and reconstructed sample information of the reference sample information; Accumulating the second distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the second distortion cost value of the current image block; It is determined whether the current image block uses a loop filtering technology based on a neural network according to a minimum distortion cost value of the first distortion cost value and the second distortion cost value of the current image block.

29. The method according to any one of claims 18 to 28, wherein: The parameter sample information also includes: candidate quantization parameters; The method further includes: determining a target quantization parameter of a current image block where the current coding tree unit is located from candidate quantization parameters according to a first distortion cost value of the current coding tree unit.

30. The method of claim 29, wherein: Also includes: Determine whether to adjust the quantization parameter of the current image block where the current coding tree unit is located, according to the target quantization parameter of the current image block where the current coding tree unit is located, and set a sixth syntax element; It is determined to adjust the quantization parameter of the current image block where the current coding tree unit is located, and a seventh syntax element is set according to the target quantization parameter of the current image block where the current coding tree unit is located.

31. The method according to claim 30, wherein: The sixth syntax element includes at least one of the following: The sixth syntax element at the picture sequence level is used to indicate whether to adjust the quantization parameters of the coding tree units in the picture sequence; The sixth syntax element at the picture level is used to indicate whether to adjust the quantization parameters of the coding tree units in the picture; The sixth syntax element at the slice level is used to indicate whether to adjust the quantization parameters of the coding tree units in the slice; The sixth syntax element at the coding tree unit level is used to indicate whether to adjust the quantization parameter of the coding tree unit; The seventh syntax element includes one of the following: The seventh syntax element at the picture sequence level is used to indicate the adjusted quantization parameters of all coding tree units in the picture sequence; The seventh syntax element at the picture level is used to indicate the adjusted quantization parameters of all coding tree units in the picture; The seventh syntax element at the slice level is used to indicate the adjusted quantization parameters of all coding tree units in the slice; The seventh syntax element at the coding tree unit level is used to indicate the adjusted quantization parameter of the coding tree unit.

32. The method according to any one of claims 21 to 31, wherein: The current image block includes at least one of the following: an image sequence where the current coding tree unit is located, an image where the current coding tree unit is located, a slice where the current coding tree unit is located, and a current image tree unit.

33. The method of claim 18, wherein: The reference sample information also includes at least one of the following: a quantization parameter, boundary strength information of the current coding tree unit, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image, a slice type of the current coding tree unit, and partition information of the current coding tree unit.

34. The method of claim 33, wherein: The candidate loop filter models include a first loop filter model and a second loop filter model; wherein, The reference sample information input by the first loop filter model also includes: quantization parameter and boundary strength information of the current coding tree unit; The reference sample information input by the second loop filtering model also includes: quantization parameter, reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the forward reference image, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the backward reference image.

35. The method of claim 34, wherein: include: When the candidate loop filter model is the second loop filter model, obtaining a first reference image in the first reference image list, and obtaining a first reference image in the second reference image list; When the first reference image in the first reference image list and the first reference image in the second reference image list are the same image, acquiring the second reference image in the first reference image list or the second reference image list; The reference image in the first reference image list is used as the forward reference image, and the reference image in the first reference image list is used as the backward reference image.

36. The method according to any one of claims 18 to 35, wherein: include: When the time domain level of the current coding tree unit is greater than or equal to the time domain level threshold, determining that the target loop filter model of the current coding tree unit is a second loop filter model; When the time domain level of the current coding tree unit is less than the time domain level threshold, a target loop filter model of the current coding tree unit is determined from candidate loop filter models based on a neural network.

37. The method according to any one of claims 18 to 36, wherein: The current coding tree unit is a maximum coding unit, or is obtained by scaling the maximum coding unit.

38. A decoding device, wherein: The decoding device comprises: A decoding unit configured to decode the bitstream and determine relevant syntax elements of a current coding tree unit; A first determination unit is configured to determine a target loop filter model of a current coding tree unit from candidate loop filter models based on a neural network according to the relevant syntax elements; A second determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information at least includes: predicted sample information and / or reconstructed sample information of the current coding tree unit; The filtering unit is configured to input the reference sample information of the current coding tree unit into the first loop filtering model for filtering, and output the filtered reconstructed sample information.

39. An encoding device, wherein: The encoding device comprises: A first determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information at least includes: predicted sample information and / or reconstructed sample information of the current coding tree unit; A filtering unit, configured to input the reference sample information of the current coding tree unit into a candidate loop filter model based on a neural network for filtering, and output the filtered reconstructed sample information; a second determining unit configured to determine a first distortion cost value of the current coding tree unit according to the original sample information of the current coding tree unit and the filtered reconstructed sample information; and determine a target loop filter model of the current coding tree unit from candidate loop filter models based on a neural network according to the first distortion cost value of the current coding tree unit; The coding unit is configured to encode relevant syntax elements of the target loop filter model of the current coding tree unit, and write the obtained coded bits into the bitstream.

40. A decoding device, wherein: The decoding device comprises: a first memory and a first processor; The first memory stores a computer program that can be run on the first processor, and the first processor implements the decoding method according to any one of claims 1 to 17 when executing the program.

41. An encoding device, wherein: The encoding device includes: a second memory and a second processor; The second memory stores a computer program that can be run on the second processor, and the second processor implements the encoding method according to any one of claims 18 to 37 when executing the program.

42. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a first processor, it implements the decoding method described in any one of claims 1 to 17; or, when the computer program is executed by a second processor, it implements the encoding method described in any one of claims 18 to 37.

Images