Encoding and decoding method and device, encoding equipment, decoding equipment and storage medium

Abstract

The invention discloses a coding and decoding method and device, coding equipment, decoding equipment and a storage medium, and the method comprises the steps: decoding a code stream, and determining related syntax elements of a current coding tree unit; determining a geometric transformation type of the current coding tree unit according to the related syntax elements; performing geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type, inputting the reference sample information of the current coding tree unit after geometric transformation into a loop filtering model based on a neural network for filtering, and outputting reconstructed sample information after filtering; and performing geometric inverse transformation on the filtered reconstructed sample information according to the geometric transformation type to obtain final reconstructed sample information of the current coding tree unit. The input information is input into the model for reasoning calculation after being subjected to geometric transformation, and the output information is subjected to inverse transformation to obtain reconstructed sample information, so that the optimal performance of the model is brought into full play, and compared with direct input, the filtering effect is better.

Description

Coding and decoding method, device, encoding device, decoding device and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of video coding and decoding technology, and in particular to a coding and decoding method, apparatus, encoding device, decoding device, and storage medium. Background Art

[0002] As demand for video display quality increases, new video applications such as high-definition and ultra-high-definition video have emerged. The Joint Video Exploration Team (JVET) of the International Organization for Standardization (ISO / IEC) and ITU-T has developed the next-generation video coding standard H.266 / Versatile Video Coding (VVC).

[0003] Neural networks have been introduced into the field of video coding and decoding. Leveraging the powerful learning capabilities of neural networks, coding and decoding tools based on neural networks often have very high coding and decoding efficiency. For example, there are intra-frame prediction methods based on neural networks, inter-frame prediction methods based on neural networks, and loop filtering methods based on neural networks. Among them, the coding performance of loop filtering methods based on neural networks is the most outstanding. However, the current loop filtering methods based on neural networks do not fully utilize the advantages of neural network models. In some coding and decoding scenarios, the loop filtering methods based on neural networks do not significantly improve the filtering effect, and may even reduce the filtering efficiency. Therefore, the loop filtering methods based on neural networks need to be optimized.

[0004] Summary of the Invention

[0005] Embodiments of the present invention provide a coding and decoding method, apparatus, coding device, decoding device, and storage medium.

[0006] In a first aspect, an embodiment of the present invention provides a decoding method, including:

[0007] Decode the code stream and determine the relevant syntax elements of the current coding tree unit;

[0008] Determining a geometric transformation type of a current coding tree unit according to the relevant syntax elements;

[0009] Determining reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0010] Performing a geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type to obtain geometrically transformed reference sample information;

[0011] Input the reference sample information after the geometric transformation of the current coding tree unit into the loop filter model based on the neural network for filtering, and output the filtered reconstructed sample information;

[0012] Performing an inverse geometric transformation on the filtered reconstructed sample information according to the geometric transformation type to obtain final reconstructed sample information of the current coding tree unit.

[0013] In a second aspect, an embodiment of the present invention provides an encoding method, including:

[0014] Determining reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0015] Performing a geometric transformation on the reference sample information of the current coding tree unit according to the candidate geometric transformation type to obtain geometrically transformed reference sample information;

[0016] Input the reference sample information after the geometric transformation of the current coding tree unit into the loop filter model based on the neural network for filtering, and output the filtered reconstructed sample information;

[0017] Performing an inverse geometric transformation on the filtered reconstructed sample information according to the candidate geometric transformation type to obtain final reconstructed sample information of the current coding tree unit;

[0018] Determining a first distortion cost value of the current coding tree unit according to original sample information of the current coding tree unit and the final reconstructed sample information;

[0019] determining a geometric transformation type of the current coding tree unit according to a first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type;

[0020] Encode the syntax elements related to the geometric transformation type of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0021] In a third aspect, an embodiment of the present invention provides a decoding device, including:

[0022] A decoding unit configured to decode the code stream and determine relevant syntax elements of a current coding tree unit;

[0023] A first determining unit configured to determine a geometric transform type of a current coding tree unit according to the relevant syntax elements;

[0024] A second determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0025] a geometric transformation unit configured to perform a geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type to obtain geometrically transformed reference sample information;

[0026] A filtering unit configured to input the reference sample information after the geometric transformation of the current coding tree unit into a loop filtering model based on a neural network for filtering, and output the filtered reconstructed sample information;

[0027] The geometric inverse transformation unit is configured to perform a geometric inverse transformation on the filtered reconstructed sample information according to the geometric transformation type to obtain final reconstructed sample information of the current coding tree unit.

[0028] In a fourth aspect, an embodiment of the present application provides an encoding device, including:

[0029] A first determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and reconstructed sample information of the current coding tree unit;

[0030] a geometric transformation unit configured to perform a geometric transformation on the reference sample information of the current coding tree unit according to the candidate geometric transformation type to obtain the reference sample information after the geometric transformation;

[0031] A filtering unit configured to input the reference sample information after the geometric transformation of the current coding tree unit into a loop filtering model based on a neural network for filtering, and output the filtered reconstructed sample information;

[0032] a geometric inverse transform unit configured to perform a geometric inverse transform on the filtered reconstructed sample information according to the candidate geometric transform type to obtain final reconstructed sample information of the current coding tree unit;

[0033] a second determining unit configured to determine a first distortion cost value of the current coding tree unit based on original sample information of the current coding tree unit and the final reconstructed sample information; and determine a geometric transformation type of the current coding tree unit based on the first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type;

[0034] The coding unit is configured to encode syntax elements related to the geometric transformation type of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0035] In a fifth aspect, an embodiment of the present application further provides a decoding device, comprising: a first memory and a first processor; the first memory stores a computer program that can be run on the first processor, and the first processor implements the decoding method of the decoder when executing the program.

[0036] In a sixth aspect, an embodiment of the present application further provides an encoding device, comprising: a second memory and a second processor; the second memory stores a computer program that can be run on the second processor, and the second processor implements the encoding method of the encoder when executing the program.

[0037] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, comprising: a computer program stored thereon, which, when executed by a first processor, implements the decoding method of the decoder; or, when executed by a second processor, implements the encoding method of the encoder.

[0038] By adopting the above technical solution, the input information is geometrically transformed and then input into the model for inference calculation, and the output information is inversely transformed to obtain reconstructed sample information, which fully exerts the optimal performance of the model and has a better filtering effect than direct input. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a block diagram of an encoder according to an embodiment of the present application;

[0040] FIG2 is a block diagram of a decoder according to an embodiment of the present application;

[0041] FIG3 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0042] FIG4 is a first schematic diagram of a geometric transformation provided in an embodiment of the present application;

[0043] FIG5 is a second schematic diagram of a geometric transformation provided in an embodiment of the present application;

[0044] FIG6 is a third schematic diagram of a geometric transformation provided in an embodiment of the present application;

[0045] FIG7 is a schematic diagram of a first neural network model provided in an embodiment of the present application;

[0046] FIG8 is a schematic diagram of an attention residual block provided in an embodiment of the present application;

[0047] FIG9 is a schematic diagram of a second neural network model provided in an embodiment of the present application;

[0048] FIG10 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0049] FIG11 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present application;

[0050] FIG12 is a schematic diagram of a specific hardware structure of a decoding device provided in an embodiment of the present application;

[0051] FIG13 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application;

[0052] FIG14 is a schematic diagram of a specific hardware structure of an encoding device provided in an embodiment of the present application;

[0053] FIG15 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present invention, the implementation of the embodiments of the present invention is described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference only and are not intended to limit the embodiments of the present invention.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first, second, and third" in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequential order where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0057] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0058] Referring to Figure 1, which shows a schematic block diagram of the composition of an encoder provided in an embodiment of the present application. As shown in Figure 1, the encoder (specifically, a "video encoder") 100 may include a transform and quantization unit 101, an intra-frame estimation unit 102, an intra-frame prediction unit 103, a motion compensation unit 104, a motion estimation unit 105, an inverse transform and inverse quantization unit 106, a filter control analysis unit 107, a filtering unit 108, an encoding unit 109, and a decoded image cache unit 110, etc., wherein the filtering unit 108 can implement deblocking filtering and sample adaptive offset (SAO) filtering, and the encoding unit 109 can implement header information encoding and context-based adaptive binary arithmetic coding (CABAC).For the input original video signal, a video coding block can be obtained by dividing it into coding tree blocks (CTUs). Then, the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transformation and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to perform intra-frame prediction on the video coding block; specifically, the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to determine the intra-frame prediction mode to be used to encode the video coding block; the motion compensation unit 104 and the motion estimation unit 105 are used to perform inter-frame prediction coding of the received video coding block relative to one or more blocks in one or more reference images to provide temporal prediction information; the motion estimation performed by the motion estimation unit 105 is a process of generating a motion vector, which can estimate the motion of the video coding block, and then the motion compensation unit 104 calculates the motion vector based on the motion determined by the motion estimation unit 105. vector performs motion compensation; after determining the intra-frame prediction mode, the intra-frame prediction unit 103 is also used to provide the selected intra-frame prediction data to the encoding unit 109, and the motion estimation unit 105 also sends the calculated motion vector data to the encoding unit 109; in addition, the inverse transform and inverse quantization unit 106 is used to reconstruct the video coding block, reconstruct the residual block in the pixel domain, and remove the block effect artifacts of the reconstructed residual block through the filter control analysis unit 107 and the filtering unit 108. The reconstructed residual block is then added to a predictive block in the frame of the decoded image buffer unit 110 to generate a reconstructed video coding block; the encoding unit 109 is used to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-frame prediction mode, and output the code stream of the video signal; and the decoded image buffer unit 110 is used to store the reconstructed video coding block for prediction reference. As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image buffer unit 110 .

[0059] Referring to Figure 2, which shows a schematic block diagram of a decoder provided in an embodiment of the present application, as shown in Figure 2, the decoder (specifically, a "video decoder") 200 includes a decoding unit 201, an inverse transform and inverse quantization unit 202, an intra-frame prediction unit 203, a motion compensation unit 204, a filtering unit 205, and a decoded image buffer unit 206. The decoding unit 201 can implement header information decoding and CABAC decoding, and the filtering unit 205 can implement deblocking filtering and SAO filtering. After the input video signal is encoded as shown in FIG1 , a code stream of the video signal is output; the code stream is input to the decoder 200 and first passes through the decoding unit 201 to obtain the decoded transform coefficients; the transform coefficients are processed by the inverse transform and inverse quantization unit 202 to generate a residual block in the pixel domain; the intra-frame prediction unit 203 can be used to generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and the data from the previously decoded block of the current image or picture; the motion compensation unit 204 determines the prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses The prediction information is used to generate a predictive block for the video decoding block being decoded; a decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 202 with the corresponding predictive block generated by the intra-frame prediction unit 203 or the motion compensation unit 204; the decoded video signal passes through the filtering unit 205 to remove blocking artifacts, thereby improving video quality; the decoded video block is then stored in the decoded image buffer unit 206, which stores reference images used for subsequent intra-frame prediction or motion compensation, and is also used for outputting the video signal, thereby obtaining the restored original video signal.

[0060] It should be noted that the method of the embodiment of the present application is mainly applied to the filtering unit 108 shown in Figure 1 and the filtering unit 205 shown in Figure 2. In other words, the embodiment of the present application can be applied to both the encoder and the decoder, or even to both the encoder and the decoder at the same time, but the embodiment of the present application is not specifically limited thereto.

[0061] In one embodiment of the present application, referring to FIG3 , a flowchart of a decoding method provided by an embodiment of the present application is shown. As shown in FIG3 , the method may include:

[0062] Step 301: Decode the code stream and determine the relevant syntax elements of the current coding tree unit;

[0063] Step 302: Determine the geometric transformation type of the current coding tree unit according to the relevant syntax elements;

[0064] The relevant syntax elements are used to indicate the geometric transform type of the current coding tree unit. The relevant syntax elements include one or more of the sequence level syntax elements, picture level syntax elements, slice level syntax elements and coding tree unit level syntax elements.

[0065] Exemplarily, in some embodiments, the relevant syntax elements include a first syntax element. Based on the first syntax element, it is determined whether the current image block where the current coding tree unit is located uses a neural network-based loop filtering technology for geometric transformation of the input. Based on the first syntax element, it is determined that the neural network-based loop filtering technology for geometric transformation of the input is used, and the type of geometric transformation is further determined; based on the first syntax element, it is determined that the neural network-based loop filtering technology for geometric transformation of the input is not used, and it is determined that the input is directly input into the neural network-based loop filtering model without geometric transformation, or other filtering technology is used.

[0066] Exemplarily, in some embodiments, the current image block includes at least one of the following: an image sequence in which the current coding tree unit is located, an image in which the current coding tree unit is located, a slice (also referred to as a "slice") in which the current coding tree unit is located, and the current image tree unit. In other words, the first syntax element is used to indicate whether the current image block uses a neural network-based loop filtering technique for geometric transformation of the input.

[0067] Exemplarily, the first syntax element includes at least one of the following: a first syntax element at an image sequence level, used to indicate whether the image sequence uses a neural network-based loop filtering technology for geometric transformation of the input; a first syntax element at an image level, used to indicate whether the image uses a neural network-based loop filtering technology for geometric transformation of the input; a first syntax element at a slice level, used to indicate whether the slice uses a neural network-based loop filtering technology for geometric transformation of the input; a first syntax element at a coding tree unit level, used to indicate whether the coding tree unit uses a neural network-based loop filtering technology for geometric transformation of the input.

[0068] In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level. In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level and a first syntax element at a picture level. In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level and a first syntax element at a slice level. In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level, a first syntax element at a picture level (or a slice level), and a first syntax element at a coding tree unit level.

[0069] Exemplarily, in some embodiments, the relevant syntax elements include a first syntax element and a second syntax element. When determining, based on the first syntax element, the neural network-based loop filtering technology that uses the input for geometric transformation of the current image block where the current coding tree unit is located, the geometric transformation type of the coding tree unit in the current image block is determined based on the second syntax element. That is, the first syntax element is used to indicate the neural network-based loop filtering technology that uses the input for geometric transformation. If the geometric transformation type includes two or more, a second syntax element needs to be added to indicate the geometric transformation type. In practical applications, the different meanings referred to by the syntax elements are represented by setting the values ​​of the syntax elements. If only one geometric transformation type is included, the first syntax element is used to indicate the neural network-based loop filtering technology that uses the input for geometric transformation, and can also be used to indicate the geometric transformation type.

[0070] Exemplarily, the second syntax element includes one of the following: a second syntax element at the image sequence level, used to indicate the geometric transformation type of all coding tree units in the image sequence; a second syntax element at the image level, used to indicate the geometric transformation type of all coding tree units in the image; a second syntax element at the slice level, used to indicate the geometric transformation type of all coding tree units in the slice; a second syntax element at the coding tree unit level, used to indicate the geometric transformation type of the coding tree unit.

[0071] Exemplarily, in some embodiments, the relevant syntax elements further include a third syntax element, and based on the third syntax element, it is determined whether the current image block uses a neural network-based loop filtering technology. Based on the third syntax element, it is determined that a neural network-based loop filtering technology is used, and subsequent relevant syntax elements are parsed to determine whether the input undergoes a geometric transformation, and the type of geometric transformation; based on the third syntax element, it is determined whether to use a neural network-based loop filtering technology, use other filtering technologies, or not use filtering technology. In some embodiments, when the third syntax element is a first preset value, it is determined that all coding tree units in the current image block do not use a neural network-based loop filtering technology; when the third syntax element is a second preset value, it is determined that all coding tree units in the current image block use a neural network-based loop filtering technology; when the third syntax element is a second preset value, it is determined that some coding tree units in the current image block use a neural network-based loop filtering technology.

[0072] Exemplarily, the third syntax element includes at least one of the following: a third syntax element at the image sequence level, used to indicate whether the image sequence uses a neural network-based loop filtering technology; a third syntax element at the image level, used to indicate whether the image uses a neural network-based loop filtering technology; a third syntax element at the slice level, used to indicate whether the slice uses a neural network-based loop filtering technology; a third syntax element at the coding tree unit level, used to indicate whether the coding tree unit uses a neural network-based loop filtering technology.

[0073] In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a picture level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a slice level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level, a third syntax element at a picture level (or a slice level), and a third syntax element at a coding tree unit level.

[0074] Exemplarily, in some embodiments, the relevant syntax elements further include a fourth syntax element, and based on the fourth syntax element, it is determined whether the current image block allows the use of a neural network-based loop filtering technique for geometric transformation of the input. Based on the fourth syntax element, it is determined that the use of a neural network-based loop filtering technique for geometric transformation of the input is allowed, and subsequent relevant syntax elements are parsed to determine whether the input is geometrically transformed and the type of geometric transformation; based on the fourth syntax element, it is determined whether the use of a neural network-based loop filtering technique for geometric transformation of the input is not allowed, and whether other filtering techniques are used, or whether no filtering technique is used.

[0075] Exemplarily, in some embodiments, the fourth syntax element includes at least one of the following: a fourth syntax element at the image sequence level, used to indicate whether the image sequence allows the use of a neural network-based loop filtering technique for geometric transformation of the input; a fourth syntax element at the image level, used to indicate whether the image allows the use of a neural network-based loop filtering technique for geometric transformation of the input; a fourth syntax element at the slice level, used to indicate whether the slice allows the use of a neural network-based loop filtering technique for geometric transformation of the input; a fourth syntax element at the coding tree unit level, used to indicate whether the coding tree unit allows the use of a neural network-based loop filtering technique for geometric transformation of the input.

[0076] In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level. In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level and a fourth syntax element at a picture level. In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level and a fourth syntax element at a slice level. In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level, a fourth syntax element at a picture level (or a slice level), and a fourth syntax element at a coding tree unit level.

[0077] In some embodiments, the relevant syntax elements further include a fifth syntax element, and based on the fifth syntax element, it is determined whether the current image block allows the use of a neural network-based loop filtering technique. Based on the fourth syntax element, it is determined that the use of a neural network-based loop filtering technique is allowed, and subsequent relevant syntax elements are parsed to determine whether the input is geometrically transformed and the type of geometric transformation; based on the fifth syntax element, it is determined whether the use of a neural network-based loop filtering technique is not allowed, and whether other filtering techniques are used, or whether no filtering technique is used.

[0078] In some embodiments, the fifth syntax element includes at least one of the following: a fifth syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the image level, used to indicate whether the image is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the slice level, used to indicate whether the slice is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the coding tree unit level, used to indicate whether the coding tree unit is allowed to use the neural network-based loop filtering technology.

[0079] In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a picture level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a slice level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level, a fifth syntax element at a picture level (or a slice level), and a fifth syntax element at a coding tree unit level.

[0080] Step 303: Determine reference sample information of the current coding tree unit; wherein the reference sample information at least includes: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0081] For example, as shown in FIG1 , each image in the input video is divided into square largest coding units (LCUs) of the same size (e.g., 128x128, 64x64, etc.). Each LCU can be divided into rectangular coding units (CUs) according to a rule. Coding units may also be divided into prediction units (PUs), transform units (TUs), etc. The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra prediction and inter prediction. Inter prediction includes motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in an image of a video, the intra prediction method is used in video coding and decoding technology to eliminate spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent images in a video, the inter prediction method is used in video coding and decoding technology to eliminate temporal redundancy between adjacent frames, thereby improving coding and decoding efficiency.

[0082] The current image is divided into blocks. Intra-frame prediction or inter-frame prediction is used for the current block to generate a prediction block for the current block. This prediction block is then used to form a prediction block (i.e., prediction sample information) for the coding tree unit. The bitstream is then parsed to obtain a quantization coefficient matrix. This matrix is ​​then dequantized and inversely transformed to obtain a residual block. The prediction block and residual block are then added together to form a reconstructed block. The reconstructed block forms the reconstructed image (i.e., reconstructed sample information) for the coding tree unit. The reconstructed image is then loop-filtered using the coding tree unit (i.e., the maximum coding unit size) as the basic processing unit to produce the decoded image.

[0083] Exemplarily, in some embodiments, the reference sample information also includes constant parameters and non-constant parameters of the current tree coding unit; the constant parameters include at least one of the following: quantization parameters, image type or slice type corresponding to the current coding tree unit; the non-constant parameters include at least one of the following: boundary strength information of the current coding tree unit, partition information of the current coding tree unit, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image.

[0084] The constant parameter can be understood as the same parameter corresponding to each pixel in the current coding tree unit. The non-constant parameter can be understood as the parameter corresponding to each pixel in the current coding tree unit is not completely the same. The geometric transformation of the reference sample information of the current coding tree unit includes the geometric transformation of all non-constant parameters in the reference sample information.

[0085] In some embodiments, the method further includes: determining, based on a sixth syntax element, whether to adjust constant parameters of a current image block in which the current coding tree unit is located; if it is determined, based on the sixth syntax element, that the constant parameters of the current image block in which the current coding tree unit is located are adjusted, determining, based on a seventh syntax element, the adjusted constant parameters of the current image block in which the current coding tree unit is located; and adjusting the constant parameters based on the adjustment parameters, and inputting the adjusted constant parameters into the neural network-based loop filtering technique. In some embodiments, the constant parameters are quantization parameters.

[0086] In some embodiments, the sixth syntax element includes at least one of the following: a sixth syntax element at the image sequence level, used to indicate whether to adjust the constant parameters of the coding tree units in the image sequence; a sixth syntax element at the image level, used to indicate whether to adjust the constant parameters of the coding tree units in the image; a sixth syntax element at the slice level, used to indicate whether to adjust the constant parameters of the coding tree units in the slice; a sixth syntax element at the coding tree unit level, used to indicate whether to adjust the constant parameters of the coding tree units.

[0087] In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a picture level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a slice level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level, a sixth syntax element at a picture level (or a slice level), and a sixth syntax element at a coding tree unit level.

[0088] The seventh syntax element includes one of the following: a seventh syntax element at the picture sequence level, used to indicate the adjusted constant parameters of all coding tree units in the picture sequence; a seventh syntax element at the picture level, used to indicate the adjusted constant parameters of all coding tree units in the picture; a seventh syntax element at the slice level, used to indicate the adjusted constant parameters of all coding tree units in the slice; a seventh syntax element at the coding tree unit level, used to indicate the adjusted constant parameters of the coding tree unit.

[0089] In some embodiments, the method further includes: determining target constant parameters for the current image block in which the current coding tree unit is located based on a sixth syntax element. That is, the sixth syntax element may directly indicate the constant parameters of the current coding tree unit. Alternatively, the sixth syntax element may indicate whether the constant parameters of the current coding tree unit are to be adjusted. If so, the seventh syntax element may indicate the adjustment parameters, and the adjusted constant parameters may be determined based on the adjustment parameters.

[0090] Step 304: performing a geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type to obtain geometrically transformed reference sample information;

[0091] Exemplarily, the geometric transformation type includes one of the following: diagonal flip, horizontal flip, vertical flip, and rotation of a preset angle.

[0092] This paper proposes a variety of geometric transformation model input schemes based on neural network loop filtering, that is, the information to be input into the neural network model is first geometrically transformed and then input into the model. After the neural network-based loop filtering model inference filtering, the output information is inversely operated to transform the samples in the output block back to their original positions.

[0093] Specifically, when the current input information needs to be horizontally flipped, the vertical coordinate can be fixed and unchanged, and the horizontal coordinate can be flipped. As shown in FIG4 , the horizontal flip is performed along the vertical center line of the image.

[0094] Assuming that the width of the original model input patch is patchWidth and the height is patchHeight, x represents the horizontal coordinate of the sample, y represents the vertical coordinate of the sample, flip_x represents the horizontal coordinate of the flipped sample, and flip_y represents the vertical coordinate of the flipped sample, the calculation formula for horizontal flipping can be as follows:

[0095] flip_x = patchWidth – x;

[0096] flip_y=y.

[0097] Horizontal flipping can also be performed along any vertical line of the image instead of along the vertical center line of the image.

[0098] Specifically, when the current input information needs to be flipped vertically, the horizontal coordinate can be fixed and unchanged, and the vertical coordinate can be flipped. As shown in FIG5 , the vertical flip is performed along the horizontal center line of the image.

[0099] As above, assuming flip_x represents the horizontal coordinate of the sample after flipping, and flip_y represents the vertical coordinate of the sample after flipping, the calculation formula for vertical flipping can be as follows:

[0100] flip_x = x;

[0101] flip_y = patchHeight – y.

[0102] Vertical flipping can also be performed along any horizontal line of the image instead of along the horizontal center line of the image.

[0103] Specifically, when the current input information needs to be flipped diagonally, the horizontal coordinate and the vertical coordinate may be interchanged. Such a diagonal flip may also be referred to as flipping along the upper left to lower right diagonal line, as shown in FIG6 .

[0104] As above, assuming that x represents the horizontal coordinate of the sample before rotation and y represents the vertical coordinate of the sample before rotation, transpose_x represents the horizontal coordinate of the sample after rotation and transpose_y represents the vertical coordinate of the sample after rotation, the calculation formula for diagonal flipping can be as follows:

[0105] transpose_x=y;

[0106] transpose_y=x.

[0107] Similarly, diagonal flipping can also be performed along the diagonal line from the upper right to the lower left of the image.

[0108] In the above three geometric transformations, patchWidth and patchHeight are the block sizes input to the model, which can be the coding tree unit size or a larger patch block after the coding tree unit is padded outward.

[0109] Specifically, when the current input information needs to be rotated, it can be rotated 90 degrees, 180 degrees, 270 degrees, etc. with the horizontal / vertical center line as the rotation axis, or 90 degrees, 180 degrees, 270 degrees, etc. with the horizontal / vertical side line as the rotation axis, or 90 degrees, 180 degrees, 270 degrees, etc. with the diagonal line as the rotation axis.

[0110] Step 305: Input the reference sample information after the geometric transformation of the current coding tree unit into the loop filtering model based on the neural network for filtering, and output the filtered reconstructed sample information;

[0111] Exploration of neural network-based loop filtering solutions primarily focuses on two forms: a multi-model switchable solution and a single-model solution. These two solutions serve as baseline solutions for neural network loop filtering. The reference software integrating these two solutions will be named the Neural Network Based Video Coding Reference Software (NNVC). All neural network-related proposals must use NNVC as a benchmark for performance and complexity comparison. It is worth noting that the basic processing unit of the neural network loop filtering solution is the coding tree unit, which is the maximum coding unit size.

[0112] The biggest difference between the first multi-model switchable solution and the second single-model solution is that the multi-model solution can provide better performance than a single model for different application scenarios or configuration conditions; however, the most obvious disadvantage of the multi-model solution is that it requires storing multiple models and requires loading a large number of models during calculation and inference.

[0113] The specific first multi-model solution is shown in Figure 7. The 2Nx2N input is usually the size of a coding tree unit, but in some specific implementations or methods, the boundary pixels are often copied outward, commonly known as the padding operation. This is because, as shown in the figure above, if the 3x3 convolutional neural network needs to perform a convolution operation on the boundary, it needs to expand outward, otherwise the convolution cannot be performed. It is worth noting that the design of the attention residual block (Attention Residual block) shown in Figure 8 may also be different in different implementations and methods. The numbers and designs here are only for the convenience of understanding the entire neural network-based loop filter model framework. The neural network-based loop filter model framework mainly consists of the input part, the main network part, and the output part. The main reasoning is shown in the figure by 3x3 convolution connected to multiple ARblocks, and finally a 3x3 convolution operation and a shuffle operation on the data position transformation.

[0114] The Attention Residual block is a modular module consisting of multiple convolutional layers, a Prelude (PReLU) activation layer, and an Attention layer. This article does not provide a detailed description of the various operational layers within the Attention Residual block. The use, number, and internal structure of these modular modules do not significantly impact the proposed techniques. It should be noted that the proposed solution applies to both the Attention Residual block shown in Figure 8 and other ResBlocks.

[0115] The input part currently mainly includes reconstructed sample data rec, predicted sample data pred, quantization parameter information QP, boundary strength information BS, etc. Different slice types, such as intra-frame I_Slice and dual reference image B_Slice, can use different models, and the input part may be different. Specifically, the I_Slice model can have an additional partition information partition as input. Different color components may have different applicable models, and the input information may also be different. Specifically, the chroma component model usually requires not only the reconstructed sample data rec of the chroma component, but also the reconstructed sample data rec of the luminance component to improve filtering performance.

[0116] As for the output part output, this part is basically the residual information res of the current coding tree unit or the reconstructed sample information rec of the current coding tree unit.

[0117] As mentioned above, for different filtering targets, such as luma and chroma, a multi-model solution can train a separate model for luma and a separate model for chroma. For example, for I_slice and B_slice, a multi-model solution can train a model for I_slice and a separate model for B_slice. Specifically, the current mainstream solution has four models, corresponding to luma I_slice, luma B_slice, chroma I_slice, and chroma B_slice.

[0118] Figure 9 shows the specific second single-model solution. While the input and filter sizes for this solution are the same as those for the first solution, the block used in this solution is clearly different: the ResBlock. As shown in the figure, the ResBlock primarily consists of a 1x1xKxK convolutional layer, a ReLU activation layer followed by a 1x1xKxK convolutional layer, and a 3x3xKxK convolutional layer. It's important to note that both the ARBlock in the first solution and the ResBlock in the second solution have skip connections, connecting the input and output within this block.

[0119] For the input part, the input types of the second scheme mainly consist of reconstructed samples rec, predicted samples pred, and three constant input base quantization parameter information BaseQP, slice-level quantization parameter information SliceQP, and slice type Slicetype. Here, "slice" Slice can be broadly understood as image level or picture level. Since the second scheme is a single model, it can process different color components and different image types. Therefore, the input part needs to input this information at one time to help the neural network better filter the current coding tree unit. Therefore, the reconstructed samples rec include luminance reconstructed samples rec and chrominance reconstructed samples rec, the predicted samples pred include luminance prediction samples pred and chrominance prediction samples pred, and the slice type Slicetype indicates that the current coding tree unit is of I_Slice type, B_Slice type, or even other types.

[0120] For the output part, in the first solution, if it is a luma model, the output is the luma reconstructed sample rec or luma residual information res, and if it is a chroma model, the output is the chroma reconstructed sample rec or chroma residual information res. The second solution is a single model, so it directly outputs the reconstructed sample rec or residual information res containing luma and chroma.

[0121] In summary, the main differences between the two solutions lie in the input part and the number of models. This document does not elaborate on the main part of the neural network.

[0122] During model training and usage, due to the generalization of the model and the fact that the model training input includes quantization parameter information, different filtering results can be obtained by adjusting these input parameters when using the model to infer the filtered coding tree units. The encoder selects the optimal parameters for these different filtering results based on the principle of minimizing rate distortion. These parameters are written into the bitstream and transmitted at the slice level or the coding tree unit (CTU) level. The decoder parses the bitstream to obtain these adjusted parameters and performs the same adjustments as the encoder, achieving the same filtering results as the encoder. This part can be summarized as model input parameter adjustment.

[0123] In addition, a scaling operation scale is usually performed on the reconstructed sample rec after filtering. The scaling factor scaling factor is obtained at the encoding end by the global minimum mean square error method or other methods that calculate a certain variance to minimize the difference between the original sample, the reconstructed sample after neural network filtering, and the reconstructed sample before filtering. The scaling factor is written into the bitstream at the slice level, the coding tree unit level CTU, or other combined division areas and transmitted to the decoding end. The decoding end obtains the same scaling factor as the encoding end by parsing the bitstream and scales the reconstructed sample rec after filtering to obtain the final reconstructed sample rec of the coding tree unit. The specific scaling operation is as follows: rec refine =(rec cnn -rec before )*scale_factor+rec before

[0124] Among them, rec refine is the reconstructed sample after scaling, rec cnn is the reconstructed sample after neural network filtering, rec before is the reconstructed sample before neural network filtering, and scale_factor is the scaling factor. It is worth noting that the above is a theoretical calculation process, and the specific implementation includes using some shift operations to replace multiplication operations.

[0125] As mentioned above, the geometric transformation of the input reference sample information can be used for all non-constant inputs, specifically, taking the second neural network loop filtering scheme introduced above as an example. As shown in Figure 9, the model input information includes reconstructed samples rec, predicted samples pred, image or slice type Slicetype, quantization parameter information BaseQP and SliceQP, of which only the reconstructed samples rec and the predicted samples pred are non-constant information, while the image or slice type slicetype, quantization parameter BaseQP and quantization parameter SliceQP are usually constants in the loop filtering based on the coding tree unit and the same constant is used for all positions. Therefore, taking the second neural network loop filtering scheme as an example, if a geometric transformation is required, it is sufficient to flip or rotate the reconstructed samples rec and the predicted samples pred.

[0126] Taking the first neural network loop filtering solution described above as an example, due to the different neural network-based loop filtering models and different input information, in addition to geometric transformations of the reconstructed samples rec and predicted samples pred, geometric transformations are also required for the boundary strength information BS and the partition information part.

[0127] It is worth noting that the input information of the model described in this article may not be comprehensive. For example, in some applications or solutions, the input of the neural network-based loop filtering model contains the coding tree unit information or other non-constant information at the same position in the reference image, and the geometric transformation method proposed in this article can also be used.

[0128] Step 306: Performing an inverse geometric transformation on the filtered reconstructed sample information according to the geometric transformation type to obtain final reconstructed sample information of the current coding tree unit.

[0129] Exemplarily, in some embodiments, the method further includes: when the time domain level of the current coding tree unit is greater than or equal to the time domain level threshold, determining the geometric transformation type of the current coding tree unit based on the relevant syntax elements; when the time domain level of the current coding tree unit is less than the time domain level threshold, determining not to use the neural network-based loop filtering technology for geometric transformation of the input.

[0130] For example, in some embodiments, the current coding tree unit is a maximum coding unit, or is obtained by scaling a maximum coding unit. That is, in the above geometric transformation, patchWidth and patchHeight are the block sizes input to the model, which can be the size of the coding tree unit or a larger patch block after padding the coding tree unit.

[0131] The method of multiple geometric transformation model inputs proposed in this paper will introduce one or more new syntax elements into the neural network-based loop filtering scheme. These syntax elements include but are not limited to the following:

[0132] The fourth syntax element of the image sequence level, sps_nnlf_geotransform_enable_flag (sequence level switch flag);

[0133] The fourth syntax element at the picture level or slice level, slice_nnlf_geotransform_enable_flag (picture level or slice level switch flag);

[0134] The first syntax element of the picture level or slice level, slice_nnlf_geotransform_flag (picture level or slice level usage flag);

[0135] The second syntax element at the picture level or slice level, slice_nnlf_geotransform_index (picture level or slice level index);

[0136] The first syntax element at the coding tree unit level, ctb_nnlf_geotransform_flag (coding tree unit level usage flag);

[0137] The second syntax element at the coding tree unit level is ctb_nnlf_geotransform_index (coding tree unit level index).

[0138] If the same or similar sps_nnlf_geotransform_enable_flag syntax element is used, the technology proposed in the embodiment of this application is applicable; otherwise, no sequence-level permission flag is required, and the current sequence is defaulted to allowing the use of the technology proposed in this article.

[0139] The decoding end parses the sequence-level flag. If sps_nnlf_enable_flag (the fifth syntax element at the image sequence level) is true, it indicates that the current bitstream allows the use of neural network-based loop filtering technology, and the subsequent decoding process needs to parse the relevant syntax elements, such as parsing the bitstream to obtain sps_nnlf_geotransform_enable_flag; otherwise, it indicates that the current bitstream does not allow the use of neural network-based loop filtering technology, and the subsequent decoding process does not need to parse the relevant syntax elements. The default relevant syntax elements are initial values ​​or false states.

[0140] 1. If sps_nnlf_enable_flag is true, the decoder parses the syntax elements related to the neural network loop filtering technology of the current image or slice, and obtains the image-level or slice-level identification bit slice_nnlf_flag (i.e., the third syntax element at the image level or slice level) based on the neural network loop filtering technology; otherwise, all the neural network loop filtering identification positions related to the technology indicated by the sequence level are set to the default values, and step 3 is executed.

[0141] If sps_nnlf_enable_flag is true and the current picture or slice allows the use of the various geometric transformation model input techniques proposed in this paper, and slice_nnlf_flag is not 0, the decoder parses the relevant syntax elements of the current picture or slice level for the techniques proposed in this paper to obtain the picture-level or slice-level flag slice_nnlf_geotransform_flag; otherwise, slice_nnlf_geotransform_flag is set to false. If slice_nnlf_geotransform_flag is true, the decoder further parses the bitstream and obtains the index value of the current picture-level or slice-level syntax element slice_nnlf_geotransform_index.

[0142] 2. If slice_nnlf_flag is 0, it means that all coding tree units in the current image or slice do not use neural network-based loop filtering technology, and execute step 3;

[0143] If slice_nnlf_flag is 1, all CTUs in the current image or slice are filtered using neural network-based loop filtering (NNLF). The ctb_nnlf_flag flag for all CTUs in the current image or slice is set to true. Filtering is performed on all CTUs in the current image or slice, loading the NNLF-based loop filtering model applicable to the current image, slice, or CTU. If slice_nnlf_geotransform_flag is true, a geometric transformation is applied to the non-constant inputs to the model. The type of transformation is determined by slice_nnlf_geotransform_index. In this embodiment, the codec has simultaneously specified that the index for horizontal flipping is 0, the index for vertical flipping is 1, and the index for diagonal flipping is 2. Based on the slice_nnlf_geotransform_index obtained from the parsed bitstream, the corresponding geometric transformation type is selected to perform geometric transformations on the reconstructed samples rec and the predicted samples pred. These transformations are then input into the model together with the quantization parameter BaseQP, the quantization parameter SliceQP, and the image or slice type SliceType for inference filtering to obtain the residual information or filtered reconstructed samples of the current coding tree unit. The output information of the model is inversely transformed to obtain the filtered reconstructed samples of the normal order of the current coding tree unit. The inverse transformation type is the same as the transformation type before input into the model. The specific method is not elaborated here.

[0144] If slice_nnlf_flag is 2, it indicates that some CTUs in the current image or slice use neural network-based loop filtering (NN-based loop filtering) while others do not. Further analysis of the ctb_nnlf_flag flag for all CTUs in the current image or slice is required. The NN-based loop filtering model corresponding to the current image, slice, or CTU is loaded, and all CTUs in the current image or slice are traversed. If ctb_nnlf_flag is true, the CTU is filtered using the loaded model; if ctb_nnlf_flag is false, the CTU is not filtered. The non-constant input to the model for filtering is geometrically transformed according to slice_nnlf_geotransform_flag and slice_nnlf_geotransform_index. The output of the model is inversely transformed to obtain the filtered, reconstructed samples of the current CTU in the normal order. The specific CTU-level filtering process is the same as described above and is not further described here.

[0145] After traversing all coding tree units in the current image or slice, the neural network-based loop filtering module ends.

[0146] 3. The decoder continues to traverse other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article, so it will not be explained in detail here.

[0147] Table 1 is a brief description of the decoding end analysis process.

[0148]

[0149]

[0150] In this embodiment, the coding tree unit level allows for various parameter adjustments, typically applied to the model's constant inputs. For example, both the neural network-based loop filtering schemes 1 and 2 described in the introduction require constant input information, including quantization parameters and the current image or slice type. This embodiment focuses on coupling this constant input parameter adjustment with the techniques presented herein, allowing for the two to be combined.

[0151] Specifically, taking the neural network-based loop filtering solution 1 as an example, the quantization parameter BaseQP of the model input is adjusted according to the preset step size / bias value / compensation value / candidate value. In this embodiment, the compensation value is used as an example, and the adjustment calculation is as follows:

[0152] FinalBaseQP=BaseQP+offset

[0153] In the above formula, FinalBaseQP is the quantization parameter BaseQP information finally input into the model, and offset is the quantization parameter compensation value (i.e., adjustment parameter), which can be 0, +5, -5, +10, and -10, etc.

[0154] If the standard text has the same or similar sps_nnlf_geotransform_flag syntax element, it is applicable to the technology proposed in this article; otherwise, it is assumed that the technology proposed in this article does not require a sequence-level permission flag, and the current sequence is assumed to allow the use of the technology proposed in this article.

[0155] The decoding end parses the sequence-level flag. If sps_nnlf_enable_flag is true, it means that the current bitstream allows the use of neural network-based loop filtering technology, and the subsequent decoding process needs to parse related syntax elements, such as parsing the bitstream to obtain sps_nnlf_geotransform_flag; otherwise, it means that the current bitstream does not allow the use of neural network-based loop filtering technology, and the subsequent decoding process does not need to parse related syntax elements. The default related syntax elements are initial values ​​or false.

[0156] 1. If sps_nnlf_enable_flag is true, the decoder parses the syntax elements related to the neural network loop filtering technology of the current image or slice, and obtains the image-level or slice-level identification bit slice_nnlf_flag based on the neural network loop filtering technology; otherwise, all the neural network loop filtering identification positions related to the technology indicated by the sequence level are set to the default values, and step 3 is executed.

[0157] If the current picture-level or slice-level flag slice_nnlf_flag is not 0, the bitstream is parsed to obtain the current picture-level or slice-level flag slice_nnlf_qpadj_flag. Otherwise (i.e., slice_nnlf_flag is 0), the current picture-level or slice-level flag slice_nnlf_qpadj_flag is set to false by default. If slice_nnlf_qpadj_flag is true, the bitstream is parsed to obtain the current picture-level or slice-level index slice_nnlf_qpadj_index.

[0158] If sps_nnlf_enable_flag is true and the current picture or slice allows the use of the various geometric transformation model input techniques proposed in this paper, and slice_nnlf_flag is not 0, the decoder parses the relevant syntax elements of the current picture or slice level for the techniques proposed in this paper to obtain the picture-level or slice-level flag slice_nnlf_geotransform_flag; otherwise, slice_nnlf_geotransform_flag is set to false. If slice_nnlf_geotransform_flag is true, the decoder further parses the bitstream and obtains the index value of the current picture-level or slice-level syntax element slice_nnlf_geotransform_index.

[0159] 2. If slice_nnlf_flag is 0, it means that all coding tree units in the current image or slice do not use neural network-based loop filtering technology, and execute step 3;

[0160] If slice_nnlf_flag is 1, all CTUs in the current image or slice are filtered using a neural network-based loop filter (NNLF) technique. The usage flag ctb_nnlf_flag for all CTUs in the current image or slice is set to true. Filtering is performed on all CTUs in the current image or slice, and the NNLF-based loop filter model applicable to the current image, slice, or CTU is loaded. If slice_nnlf_qpdj_flag is true, the quantization parameter BaseQP of the model input is adjusted based on the index of slice_nnlf_qpadj_index; otherwise, no adjustment is required. The adjustment process is the same as on the encoder side. If slice_nnlf_qpadj_index is 0, the quantization parameter input to the model is BaseQP-5; if slice_nnlf_qpadj_index is 1, the quantization parameter input to the model is BaseQP+5. In addition, if slice_nnlf_geotransform_flag is true, it is necessary to perform a geometric transformation on the non-constant input of the model, and the type of transformation is determined by slice_nnlf_geotransform_index. In this embodiment, the codec has simultaneously stipulated that the index of horizontal flip is 0, the index of vertical flip is 1, and the index of diagonal flip is 2. The corresponding geometric transformation type is selected according to the slice_nnlf_geotransform_index obtained by parsing the code stream to perform a geometric transformation on the reconstructed sample rec and the predicted sample pred, and then input them together with the quantization parameter BaseQP, the quantization parameter SliceQP and the image or slice type SliceType into the model for inference filtering to obtain the residual information of the current coding tree unit or the filtered reconstructed sample. The output information of the model is inversely transformed to obtain the filtered reconstructed samples of the normal order of the current coding tree unit. The inverse transformation type is the same as the transformation type before inputting into the model. The specific method is not elaborated here.

[0161] If slice_nnlf_flag is 2, it indicates that some CTUs in the current image or slice use neural network-based loop filtering (NN-based loop filtering) while others do not. Further analysis of the CTU-level usage flag (ctb_nnlf_flag) for all CTUs in the current image or slice is required. The NN-based loop filtering model corresponding to the current image, slice, or CTU is loaded, and all CTUs in the current image or slice are traversed. If ctb_nnlf_flag is true, the loaded model is used to filter the CTU; if ctb_nnlf_flag is false, the CTU is not filtered. If slice_nnlf_qpdj_flag is true, the quantization parameter (BaseQP) of the model input is adjusted based on the index (slice_nnlf_qpadj_index); otherwise, the quantization parameter (BaseQP) does not need to be adjusted. The adjustment process is the same as that on the encoding side. If the index value of slice_nnlf_qpadj_index is 0, BaseQP-5 is the quantization parameter input of the model; if slice_nnlf_qpadj_index is 1, BaseQP+5 is the quantization parameter input of the model. In addition, the non-constant information input to the model for filtering is geometrically transformed according to slice_nnlf_geotransform_flag and slice_nnlf_geotransform_index. The output information of the model is inversely transformed by the same type of geometric transformation to obtain the filtered reconstructed samples in the normal order of the current coding tree unit. The specific coding tree unit-level filtering process is the same as above and will not be repeated here.

[0162] After traversing all coding tree units in the current image or slice, the neural network-based loop filtering module ends.

[0163] 3. The decoder continues to traverse other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article, so it will not be explained in detail here.

[0164] Table 2 is a brief description of the decoding end analysis process

[0165]

[0166]

[0167] The present application also provides an encoding method. In one embodiment of the present application, see Figure 10, which shows a schematic flow chart of an encoding method provided by the present application. As shown in Figure 10, the method may include:

[0168] Step 1001: Determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0169] The current image is divided into blocks. Intra-frame prediction or inter-frame prediction is used for the current block to generate a prediction block for the current block. This prediction block is then used to form a prediction block (i.e., prediction sample information) for the coding tree unit. The bitstream is then parsed to obtain a quantization coefficient matrix. This matrix is ​​then dequantized and inversely transformed to obtain a residual block. The prediction block and residual block are then added together to form a reconstructed block. The reconstructed block forms the reconstructed image (i.e., reconstructed sample information) for the coding tree unit. The reconstructed image is then loop-filtered using the coding tree unit (i.e., the maximum coding unit size) as the basic processing unit to produce the decoded image.

[0170] Step 1002: performing a geometric transformation on the reference sample information of the current coding tree unit according to the candidate geometric transformation type to obtain the reference sample information after the geometric transformation;

[0171] Here, the candidate geometric transformation types include at least one geometric transformation type. Exemplarily, the geometric transformation type includes one of the following: diagonal flip, horizontal flip, vertical flip, and rotation of a preset angle.

[0172] Step 1003: inputting the reference sample information after the geometric transformation of the current coding tree unit into a loop filtering model based on a neural network for filtering, and outputting the filtered reconstructed sample information;

[0173] Step 1004: performing an inverse geometric transformation on the filtered reconstructed sample information according to the candidate geometric transformation type to obtain final reconstructed sample information of the current coding tree unit;

[0174] Step 1005: determining a first distortion cost value of the current coding tree unit according to the original sample information of the current coding tree unit and the final reconstructed sample information;

[0175] Step 1006: Determine a geometric transformation type of the current coding tree unit according to a first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type;

[0176] Exemplarily, in some embodiments, determining the geometric transformation type of the current coding tree unit based on the first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type includes: accumulating the first distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the first distortion cost value of the current image block; and determining the geometric transformation type of the current coding tree unit based on the first distortion cost value of the current image block corresponding to the candidate geometric transformation type. In some embodiments, the first distortion cost value may be a rate-distortion cost value.

[0177] Exemplarily, the current image block includes at least one of the following: a current image sequence, a current image, a current slice, and a current coding tree unit. That is, the current image block is the current image sequence, the current image, or the current slice, and the geometric transformation type of the current image block is determined based on the accumulated value of the first distortion cost values ​​of all coding tree units in the current image block. The current image block is the current coding tree unit, and the geometric transformation type of the current coding tree unit is determined based on the first distortion cost value of the current coding tree unit.

[0178] Exemplarily, in some embodiments, the method further includes: inputting the reference sample information of the current coding tree unit into a loop filtering model based on a neural network for filtering, and outputting the filtered reconstructed sample information; determining the second distortion cost value of the current coding tree unit based on the original sample information of the current coding tree unit and the filtered reconstructed sample information; accumulating the second distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the second distortion cost value of the current image block; determining the third distortion cost value of the current coding tree unit based on the original sample information and the reconstructed sample information of the current coding tree unit; accumulating the third distortion cost value of all coding tree units in the current image block where the current coding tree unit is located Accumulating the distortion cost values ​​to determine a third distortion cost value of the current image block; using the minimum distortion cost value among the first distortion cost value, the second distortion cost value, and the third distortion cost value of the current coding tree unit as the fourth distortion cost value of the current coding tree unit; accumulating the fourth distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the fourth distortion cost value of the current image block; and determining whether a neural network-based loop filtering technology for performing geometric transformation of an input is used in the current image block, and whether a neural network-based loop filtering technology is used in the current image block, based on the minimum distortion cost value among the first distortion cost value, the second distortion cost value, the third distortion cost value, and the fourth distortion cost value of the current image block.

[0179] Step 1007: Encode syntax elements related to the geometric transformation type of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0180] The relevant syntax elements are used to indicate the geometric transform type of the current coding tree unit. The relevant syntax elements include one or more of the sequence level syntax elements, picture level syntax elements, slice level syntax elements and coding tree unit level syntax elements.

[0181] Exemplarily, in some embodiments, the relevant syntax elements include a first syntax element. The first syntax element is set based on whether a neural network-based loop filtering technique for geometrically transforming the input is used for the current image block where the current coding tree unit is located. Based on the first syntax element, it is determined whether the neural network-based loop filtering technique for geometrically transforming the input is used, and subsequent relevant syntax elements are parsed to determine the type of geometric transform; based on the first syntax element, it is determined whether the neural network-based loop filtering technique for geometrically transforming the input is not used, the input is not geometrically transformed, or another filtering technique is used.

[0182] Exemplarily, in some embodiments, the current image block includes at least one of the following: an image sequence where the current coding tree unit is located, an image where the current coding tree unit is located, a slice where the current coding tree unit is located, and a current image tree unit.

[0183] Exemplarily, the first syntax element includes at least one of the following: a first syntax element at an image sequence level, used to indicate whether the image sequence uses a neural network-based loop filtering technology for geometric transformation of the input; a first syntax element at an image level, used to indicate whether the image uses a neural network-based loop filtering technology for geometric transformation of the input; a first syntax element at a slice level, used to indicate whether the slice uses a neural network-based loop filtering technology for geometric transformation of the input; a first syntax element at a coding tree unit level, used to indicate whether the coding tree unit uses a neural network-based loop filtering technology for geometric transformation of the input.

[0184] In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level. In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level and a first syntax element at a picture level. In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level and a first syntax element at a slice level. In some embodiments, the first syntax element comprises a first syntax element at a picture sequence level, a first syntax element at a picture level (or a slice level), and a first syntax element at a coding tree unit level.

[0185] Exemplarily, in some embodiments, the relevant syntax elements include a first syntax element and a second syntax element. When determining that the current image block where the current coding tree unit is located uses a neural network-based loop filtering technology for geometric transformation of the input, the second syntax element is set according to the geometric transformation type of the coding tree unit in the current image block. That is, the first syntax element is used to indicate whether to use a neural network-based loop filtering technology for geometric transformation of the input. If the geometric transformation type includes two or more, a second syntax element needs to be added to indicate the geometric transformation type. In actual applications, the different meanings referred to by the syntax elements are represented by setting the values ​​of the syntax elements.

[0186] Exemplarily, the second syntax element includes one of the following: a second syntax element at the image sequence level, used to indicate the geometric transformation type of all coding tree units in the image sequence; a second syntax element at the image level, used to indicate the geometric transformation type of all coding tree units in the image; a second syntax element at the slice level, used to indicate the geometric transformation type of all coding tree units in the slice; a second syntax element at the coding tree unit level, used to indicate the geometric transformation type of the coding tree unit.

[0187] Exemplarily, in some embodiments, the relevant syntax elements further include a third syntax element, which is set based on whether the current image block uses a neural network-based loop filtering technique. Based on the third syntax element, it is determined whether the neural network-based loop filtering technique is used, and subsequent relevant syntax elements are parsed to determine whether the input is geometrically transformed and the type of geometric transformation. Based on the third syntax element, it is determined whether to not use the neural network-based loop filtering technique, to use another filtering technique, or to use no filtering technique.

[0188] Exemplarily, the third syntax element includes at least one of the following: a third syntax element at the image sequence level, used to indicate whether the image sequence uses a neural network-based loop filtering technology; a third syntax element at the image level, used to indicate whether the image uses a neural network-based loop filtering technology; a third syntax element at the slice level, used to indicate whether the slice uses a neural network-based loop filtering technology; a third syntax element at the coding tree unit level, used to indicate whether the coding tree unit uses a neural network-based loop filtering technology.

[0189] In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a picture level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level and a third syntax element at a slice level. In some embodiments, the third syntax element comprises a third syntax element at a picture sequence level, a third syntax element at a picture level (or a slice level), and a third syntax element at a coding tree unit level.

[0190] Exemplarily, in some embodiments, the relevant syntax elements further include a fourth syntax element, and the fourth syntax element is set based on whether the current image block allows the use of a neural network-based loop filtering technique for geometric transformation of the input. The fourth syntax element determines whether the neural network-based loop filtering technique for geometric transformation of the input is allowed, and subsequent relevant syntax elements are parsed to determine whether the input is geometrically transformed and the type of geometric transformation. The fourth syntax element determines whether the neural network-based loop filtering technique for geometric transformation of the input is not allowed, whether to use another filtering technique, or whether to use no filtering technique.

[0191] Exemplarily, in some embodiments, the fourth syntax element includes at least one of the following: a fourth syntax element at the image sequence level, used to indicate whether the image sequence allows the use of a neural network-based loop filtering technique for geometric transformation of the input; a fourth syntax element at the image level, used to indicate whether the image allows the use of a neural network-based loop filtering technique for geometric transformation of the input; a fourth syntax element at the slice level, used to indicate whether the slice allows the use of a neural network-based loop filtering technique for geometric transformation of the input; a fourth syntax element at the coding tree unit level, used to indicate whether the coding tree unit allows the use of a neural network-based loop filtering technique for geometric transformation of the input.

[0192] In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level. In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level and a fourth syntax element at a picture level. In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level and a fourth syntax element at a slice level. In some embodiments, the fourth syntax element comprises a fourth syntax element at a picture sequence level, a fourth syntax element at a picture level (or a slice level), and a fourth syntax element at a coding tree unit level.

[0193] In some embodiments, the relevant syntax elements further include a fifth syntax element, which is set based on whether the current image block allows the use of a neural network-based loop filtering technique. Based on the fourth syntax element, it is determined that the neural network-based loop filtering technique is allowed, and subsequent relevant syntax elements are parsed to determine whether the input is geometrically transformed and the type of geometric transformation. Based on the fifth syntax element, it is determined whether the neural network-based loop filtering technique is not allowed, and whether other filtering techniques are used, or whether no filtering technique is used.

[0194] In some embodiments, the fifth syntax element includes at least one of the following: a fifth syntax element at the image sequence level, used to indicate whether the image sequence is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the image level, used to indicate whether the image is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the slice level, used to indicate whether the slice is allowed to use the neural network-based loop filtering technology; a fifth syntax element at the coding tree unit level, used to indicate whether the coding tree unit is allowed to use the neural network-based loop filtering technology.

[0195] In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a picture level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level and a fifth syntax element at a slice level. In some embodiments, the fifth syntax element comprises a fifth syntax element at a picture sequence level, a fifth syntax element at a picture level (or a slice level), and a fifth syntax element at a coding tree unit level.

[0196] Exemplarily, in some embodiments, the reference sample information also includes constant parameters and non-constant parameters of the current tree coding unit; the constant parameters include at least one of the following: quantization parameters, image type or slice type corresponding to the current coding tree unit; the non-constant parameters include at least one of the following: boundary strength information of the current coding tree unit, partition information of the current coding tree unit, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image.

[0197] The constant parameter can be understood as the same parameter corresponding to each pixel in the current coding tree unit. The non-constant parameter can be understood as the parameter corresponding to each pixel in the current coding tree unit is not completely the same. The geometric transformation of the reference sample information of the current coding tree unit includes the geometric transformation of all non-constant parameters in the reference sample information.

[0198] In some embodiments, the reference sample information also includes: candidate constant parameters of the current coding tree unit; the method also includes: determining the target constant parameters of the current image block where the current coding tree unit is located based on the first distortion cost value of the current image block corresponding to the candidate geometric transformation type.

[0199] In some embodiments, the method further includes: determining whether to adjust the constant parameters of the current image block where the current coding tree unit is located based on the target constant parameters of the current image block where the current coding tree unit is located, and setting a sixth syntax element; determining to adjust the constant parameters of the current image block where the current coding tree unit is located, and setting a seventh syntax element based on the target constant parameters of the current image block where the current coding tree unit is located.

[0200] Here, the target constant parameter can be understood as the constant parameter corresponding to the minimum distortion cost value determined from the candidate constant parameters. When the target constant parameter differs from the original constant parameters of the current image block, the original constant parameters of the current image block are determined to be adjusted; otherwise, no adjustment is made. In some embodiments, the value of the seventh syntax element can be set based on the difference between the target constant parameter and the original constant parameters of the current image block.

[0201] In some embodiments, the sixth syntax element includes at least one of the following: a sixth syntax element at the image sequence level, used to indicate whether to adjust the constant parameters of the coding tree units in the image sequence; a sixth syntax element at the image level, used to indicate whether to adjust the constant parameters of the coding tree units in the image; a sixth syntax element at the slice level, used to indicate whether to adjust the constant parameters of the coding tree units in the slice; a sixth syntax element at the coding tree unit level, used to indicate whether to adjust the constant parameters of the coding tree units.

[0202] In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a picture level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level and a sixth syntax element at a slice level. In some embodiments, the sixth syntax element comprises a sixth syntax element at a picture sequence level, a sixth syntax element at a picture level (or a slice level), and a sixth syntax element at a coding tree unit level.

[0203] The seventh syntax element includes one of the following: a seventh syntax element at the picture sequence level, used to indicate the adjusted constant parameters of all coding tree units in the picture sequence; a seventh syntax element at the picture level, used to indicate the adjusted constant parameters of all coding tree units in the picture; a seventh syntax element at the slice level, used to indicate the adjusted constant parameters of all coding tree units in the slice; a seventh syntax element at the coding tree unit level, used to indicate the adjusted constant parameters of the coding tree unit.

[0204] In some embodiments, the method further comprises: setting a sixth syntax element according to a target constant parameter of a current image block where the current coding tree unit is located.

[0205] In some embodiments, the method further includes: when the time domain level of the current coding tree unit is greater than or equal to the time domain level threshold, determining the geometric transformation type of the current coding tree unit according to the first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type; when the time domain level of the current coding tree unit is less than the time domain level threshold, determining not to use the neural network-based loop filtering technology for input geometric transformation.

[0206] For example, in some embodiments, the current coding tree unit is a maximum coding unit, or is obtained by scaling a maximum coding unit. That is, in the above geometric transformation, patchWidth and patchHeight are the block sizes input to the model, which can be the size of the coding tree unit or a larger patch block after padding the coding tree unit.

[0207] The coding tree unit enters the neural network loop filter module and obtains the neural network-based loop filter enablement flag, namely sps_nnlf_enable_flag. If this flag is true, the neural network-based loop filter technology is enabled; if it is false, the neural network-based loop filter technology is not enabled. The sequence-level enablement flag needs to be written into the bitstream when encoding the video sequence. Here, the technology in this article can add a sequence-level enablement flag, such as the new syntax element sps_nnlf_geotransform_enable_flag. The naming of this syntax element here is mainly for ease of understanding and textual content. It can be modified in actual applications and standard texts, but its semantic content should be consistent or similar. Both indicate that if this flag is true, the neural network-based loop filter geometric transformation model input-related technology proposed in this article is enabled; if this flag is false, the neural network-based loop filter geometric transformation model input-related technology proposed in this article is not enabled.

[0208] 1. If the flag bit for allowing the use of neural network-based loop filtering is true, the encoder attempts the neural network-based loop filtering technology, that is, executes 2. If the flag bit for allowing the use of neural network-based loop filtering is false, the encoder does not attempt the neural network-based loop filtering technology, that is, skips 2 and directly executes 3.

[0209] 2. Initialize the neural network-based loop filtering technology and load the neural network model suitable for the current image.

[0210] a. Calculate the cost of the reconstruction sample

[0211] The encoder calculates the cost information without using the neural network loop filtering technology. That is, it uses the reconstructed samples of the coding tree unit to be used as network input and the original image samples of the coding tree unit to calculate and record the rate-distortion cost value. The rate-distortion cost value mainly consists of the absolute error and the number of bits consumed. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice and accumulates and records it to obtain costRec. The costRec cost information represents the accumulated cost of all coding tree units in the current image or slice.

[0212] b. Filter and calculate the cost of various geometric transformations after inputting the model

[0213] The encoding end attempts to use a neural network-based loop filtering technique, using the reconstructed sample rec, predicted sample pred, quantization parameter BaseQP, quantization parameter SliceQP, and image or slice type SliceType of the current coding tree unit to input into the loaded model for inference. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost of the filtered reconstructed sample of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice, records and accumulates it to obtain costOrgM. This costOrgM represents the accumulated cost of all coding tree units in the current image or slice;

[0214] If the current image or slice type allows the use of the geometric transformation model input technology proposed in this paper, continue to execute the remaining steps; otherwise, skip the remaining steps, assign costOrgM to costCnn, and execute 2.c;

[0215] The encoding end continues to try the neural network-based loop filtering technology, horizontally flips the reconstructed samples rec and predicted samples pred of the current coding tree unit, and inputs them into the loaded model together with the quantization parameter BaseQP, quantization parameter SliceQP and image / slice type SliceType information for inference. The neural network loop filtering model outputs the reconstructed samples of the current coding tree unit, calculates the rate-distortion cost of the reconstructed samples after filtering of the current coding tree unit and the original image samples, and records them. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice, records and accumulates them to obtain costHorflipM. This costHorflipM also represents the accumulated cost of all coding tree units in the current image or slice;

[0216] Similar to the above steps, the cumulative costs of all coding tree units in the current image or slice after vertical flipping and diagonal flipping can be filtered and calculated. The names here are the same as above, namely costVerflipM and costTranspM respectively.

[0217] Compare the sizes of costOrgM, costHorflipM, costVerflipM, and costTranspM. In this embodiment, assume that the index of horizontal flip is 0, the index of vertical flip is 1, and the index of diagonal flip is 2. If costOrgM is minimum, it means that the original filtering without using the geometric transformation model input has the best overall performance in the current image or slice, and then slice_nnlf_geotransform_flag is set to false; otherwise, it means that the model input after other geometric transformations performs better, and slice_nnlf_geotransform_flag is set to true, and the index corresponding to the minimum geometric transformation type is assigned to slice_nnlf_geotransform_index.

[0218] The minimum cost value in the above comparison is assigned to costCnn, which serves as the minimum cost information for all coding tree units at the image level or slice level to use the neural network-based loop filtering.

[0219] c. Final information of decision neural network loop filter module

[0220] The encoder is experimenting with optimizing the selection at the coding tree unit level. In the second round of attempts at neural network loop filtering (NNLF), the encoder defaulted to using NNLF for all coding tree units in the current image. This was controlled by a single image- or slice-level flag, slice_nnlf_flag, while the ctb_nnlf_flag was not required at the coding tree unit level. Now, an attempt is being made to switch the NNLF at the coding tree unit level, with each NNLF having its own flag.

[0221] The encoder traverses the coding tree units (CTUs) and compares the cost information of each CTU before and after filtering using the neural network loop filtering technique. If the unfiltered cost information is smaller, the CTU usage flag (ctb_nnlf_flag) is set to false; otherwise, the CTU usage flag (ctb_nnlf_flag) is set to true. After traversing all CTUs, the encoder accumulates the minimum cost information of each CTU to obtain the rate-distortion cost value (costCtu) for the current image or slice. This costCtu represents the cumulative value of the minimum cost for all CTUs in the current image or slice. In addition, if the current image or slice allows the use of the multiple geometric transformation model input technique proposed in this paper, the neural network loop filter model input for all CTUs in the current image or slice is determined by slice_nnlf_geotransform_flag and slice_nnlf_geotransform_index. If the current image or slice does not allow the use of the multiple geometric transformation model input technique proposed in this paper, the input of all CTUs in the current image or slice uses the original untransformed samples.

[0222] If costRec is the minimum, it means that the current image or slice is better not to use the neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 0 and written into the bitstream for transmission to the decoder, indicating that all coding tree units in the current image or slice do not use the neural network loop filtering technology, and the coding tree unit-level flag does not need to be written into the bitstream;

[0223] If costCnn is the smallest, it means that it is better to use neural network loop filtering technology for all coding tree units in the current image or slice. The image-level or slice-level flag slice_nnlf_flag is set to 1 and written into the bitstream for transmission to the decoder. The coding unit-level flag ctb_nnlf_flag does not need to be written into the bitstream. In addition, if the current image or slice allows the use of the multiple geometric model input technologies proposed in this paper, slice_nnlf_geotransform_flag needs to be written into the input bitstream for transmission to the decoder. If slice_nnlf_geotransform_flag is true, slice_nnlf_geotransform_index also needs to be written into the bitstream for transmission to the decoder.

[0224] If costCtu is the minimum, it means that not all coding tree units in the current image or slice use the neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 2 and written into the code stream for transmission to the decoder. It is also necessary to write the neural network loop filtering technology usage flag ctb_nnlf_flag of each coding tree unit into the code stream. The true or false status of this flag has been described and determined previously. In addition, if the current image or slice allows the use of multiple geometric model input technologies proposed in this article, it is necessary to write slice_nnlf_geotransform_flag into the input code stream for transmission to the decoder. If slice_nnlf_geotransform_flag is true, it is also necessary to write slice_nnlf_geotransform_index into the code stream for transmission to the decoder.

[0225] 3. The encoder continues to try other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article and will not be explained in detail here.

[0226] In this embodiment, the coding tree unit level allows for various parameter adjustments, typically applied to the model's constant inputs. For example, both the neural network-based loop filtering schemes 1 and 2 described in the introduction require constant input information, including quantization parameters and the current image or slice type. This embodiment focuses on coupling this constant input parameter adjustment with the techniques presented herein, allowing for the two to be combined.

[0227] Specifically, taking the neural network-based loop filtering solution 1 as an example, the quantization parameter BaseQP of the model input is adjusted according to the preset step size / bias value / compensation value / candidate value. In this embodiment, the compensation value is used as an example, and the adjustment calculation is as follows:

[0228] FinalBaseQP=BaseQP+offset

[0229] In the above formula, FinalBaseQP is the quantization parameter BaseQP information finally input into the model, and offset is the quantization parameter compensation value, which can be 0, +5, -5, +10, -10, etc.

[0230] For example, the newly added syntax elements are as follows: the sixth syntax element at the picture level or slice level, slice_nnlf_qpadj_flag (picture level or slice level QP adjustment flag); the seventh syntax element at the picture level or slice level, slice_nnlf_qpadj_index (picture level or slice level QP adjustment flag).

[0231] The encoder predicts the current image and obtains a prediction block for each coding unit. The residual of the coding unit is obtained by subtracting the original image block from the prediction block. The residual is transformed using various transform modes to obtain frequency-domain residual coefficients. This residual is then quantized and inversely quantized, and inversely transformed to obtain distorted residual information. This distorted residual information is superimposed on the prediction block to obtain a reconstructed block. The term "reconstructed block" here generally refers to a coding unit consisting of reconstructed samples obtained by superimposing the predicted samples with the inversely transformed and inversely quantized residual samples. Subsequently, the loop filter module filters the image at the coding tree unit level, where the proposed technology is applied.

[0232] The coding tree unit enters the neural network loop filter module and obtains the neural network-based loop filter enablement flag, namely sps_nnlf_enable_flag. If this flag is true, the neural network-based loop filter technology is enabled; if it is false, the neural network-based loop filter technology is not enabled. The sequence-level enablement flag needs to be written into the bitstream when encoding the video sequence. Here, the technology in this article can add a sequence-level enablement flag, such as the new syntax element sps_nnlf_geotransform_enable_flag. The naming of this syntax element here is mainly for ease of understanding and textual content. It can be modified in actual applications and standard texts, but its semantic content should be consistent or similar. Both indicate that if this flag is true, the neural network-based loop filter geometric transformation model input-related technology proposed in this article is enabled; if this flag is false, the neural network-based loop filter geometric transformation model input-related technology proposed in this article is not enabled.

[0233] 1. If the flag bit for allowing the use of neural network-based loop filtering is true, the encoder attempts the neural network-based loop filtering technology, that is, executes 2. If the flag bit for allowing the use of neural network-based loop filtering is false, the encoder does not attempt the neural network-based loop filtering technology, that is, skips 2 and directly executes 3.

[0234] 2. Initialize the neural network-based loop filtering technology and load the neural network model suitable for the current image.

[0235] a. Calculate the cost of the reconstruction sample

[0236] The encoder calculates the cost information without using the neural network loop filtering technology. That is, it uses the reconstructed samples of the coding tree unit to be used as network input and the original image samples of the coding tree unit to calculate and record the rate-distortion cost value. The rate-distortion cost value mainly consists of the absolute error and the number of bits consumed. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice and accumulates and records it to obtain costRec. The costRec cost information represents the accumulated cost of all coding tree units in the current image or slice.

[0237] b. Filter and calculate the cost of various geometric transformations after inputting the model

[0238] The encoding end attempts to use a neural network-based loop filtering technique, using the reconstructed sample rec, predicted sample pred, quantization parameter BaseQP, quantization parameter SliceQP, and image or slice type SliceType of the current coding tree unit to input into the loaded model for inference. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost of the filtered reconstructed sample of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice, records and accumulates it to obtain costOrgQP0. This costOrgQP0 represents the accumulated cost of all coding tree units in the current image or slice;

[0239] The encoding end continues to try the neural network-based loop filtering technology, adjusts the constant information input to the model, and filters all the coding tree units in the current image or slice. Use the reconstructed sample rec, predicted sample pred, compensated quantization parameter FinalBaseQP (BaseQP-5), quantization parameter SliceQP and image or slice type SliceType of the coding tree unit to be filtered to input into the loaded model for inference filtering. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost value of the reconstructed sample after filtering of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all the coding tree units in the current image or slice, records and accumulates it to obtain costOrgQP1. The costOrgQP1 represents the accumulated cost of all the coding tree units in the current image or slice;

[0240] The encoding end continues to try the neural network-based loop filtering technology, adjusts the constant information input to the model, and filters all the coding tree units in the current image or slice. Use the reconstructed sample rec, predicted sample pred, compensated quantization parameter FinalBaseQP (BaseQP+5), quantization parameter SliceQP and image or slice type SliceType of the coding tree unit to be filtered to input into the loaded model for inference filtering. The neural network loop filtering model outputs the reconstructed sample of the current coding tree unit, calculates the rate-distortion cost value of the reconstructed sample after filtering of the current coding tree unit and the original image sample, and records it. The encoder traverses and calculates the cost information of all the coding tree units in the current image or slice, records and accumulates it to obtain costOrgQP2. The costOrgQP2 represents the accumulated cost of all the coding tree units in the current image or slice;

[0241] Compare costOrgQP0, costOrgQP1, and costOrgQP2. If costOrgQP0 is the smallest, set slice_nnlf_qpadj_flag to false to indicate that parameter adjustment is not used for the current image or slice; otherwise, set slice_nnlf_qpadj_flag to true to indicate that parameter adjustment is used for the current image or slice. Additionally, set slice_nnlf_qpadj_index based on the size of costOrgQP1 and costOrgQP2. If costOrgQP1 is smaller than or equal to costOrgQP2, set slice_nnlf_qpadj_index to 0; otherwise, set slice_nnlf_qpadj_index to 1.

[0242] Assign the minimum value of costOrgQP0, costOrgQP1 and costOrgQP2 to costOrgM.

[0243] If the current image or slice type allows the use of the geometric transformation model input technology proposed in this paper, continue to execute the remaining steps; otherwise, skip the remaining steps, assign costOrgM to costCnn, and execute 2.c;

[0244] The encoding end continues to try the neural network-based loop filtering technology, horizontally flips the reconstructed samples rec and predicted samples pred of the current coding tree unit, and inputs them into the loaded model together with the quantization parameter BaseQP, quantization parameter SliceQP and image / slice type SliceType information for inference. The neural network loop filtering model outputs the reconstructed samples of the current coding tree unit, calculates the rate-distortion cost of the reconstructed samples after filtering of the current coding tree unit and the original image samples, and records them. The encoder traverses and calculates the cost information of all coding tree units in the current image or slice, records and accumulates them to obtain costHorflipM. The costHorflipM also represents the accumulated cost of all coding tree units in the current image or slice; the quantization parameter information BaseQP input into the model needs to be adjusted according to slice_nnlf_qpadj_flag and slice_nnlf_qpadj_index;

[0245] Similar to the above steps, the cumulative costs of all coding tree units in the current image or slice after vertical flipping and diagonal flipping can be filtered and calculated. The names here are the same as above, namely costVerflipM and costTranspM respectively.

[0246] Compare the sizes of costOrgM, costHorflipM, costVerflipM, and costTranspM. In this embodiment, assume that the index of horizontal flip is 0, the index of vertical flip is 1, and the index of diagonal flip is 2. If costOrgM is minimum, it means that the original filtering without using the geometric transformation model input has the best overall performance in the current image or slice, and then slice_nnlf_geotransform_flag is set to false; otherwise, it means that the model input after other geometric transformations performs better, and slice_nnlf_geotransform_flag is set to true, and the index corresponding to the minimum geometric transformation type is assigned to slice_nnlf_geotransform_index.

[0247] The minimum cost value in the above comparison is assigned to costCnn, which serves as the minimum cost information for all coding tree units at the image level or slice level to use the neural network-based loop filtering.

[0248] c. Final information of decision neural network loop filter module

[0249] The encoder is experimenting with optimizing the selection at the coding tree unit level. In the second round of attempts at neural network loop filtering (NNLF), the encoder defaulted to using NNLF for all coding tree units in the current image. This was controlled by a single image- or slice-level flag, slice_nnlf_flag, while the ctb_nnlf_flag was not required at the coding tree unit level. Now, an attempt is being made to switch the NNLF at the coding tree unit level, with each NNLF having its own flag.

[0250] The encoder traverses the coding tree units (CTUs) and compares the cost information of each CTU before and after filtering using the neural network loop filtering technique. If the unfiltered cost information is smaller, the CTU usage flag (ctb_nnlf_flag) is set to false; otherwise, the CTU usage flag (ctb_nnlf_flag) is set to true. After traversing all CTUs, the encoder accumulates the minimum cost information of each CTU to obtain the rate-distortion cost value (costCtu) for the current image or slice. This costCtu represents the cumulative value of the minimum cost for all CTUs in the current image or slice. In addition, if the current image or slice allows the use of the multiple geometric transformation model input technique proposed in this paper, the neural network loop filter model input for all CTUs in the current image or slice is determined by slice_nnlf_geotransform_flag and slice_nnlf_geotransform_index. If the current image or slice does not allow the use of the multiple geometric transformation model input technique proposed in this paper, the input of all CTUs in the current image or slice uses the original untransformed samples. In addition, the model input quantization parameter BaseQP information of the filtering process also needs to be determined according to slice_nnlf_qpadj_flag and slice_nnlf_qpadj_index determined in 2.b.

[0251] If costRec is the minimum, it means that the current image or slice is better not to use the neural network loop filtering technology. The image-level or slice-level flag slice_nnlf_flag is set to 0 and written into the bitstream for transmission to the decoder, indicating that all coding tree units in the current image or slice do not use the neural network loop filtering technology, and the coding tree unit-level flag does not need to be written into the bitstream;

[0252] If costCnn is the smallest, it means that it is better to use neural network loop filtering technology for all coding tree units in the current image or slice. The image-level or slice-level flag slice_nnlf_flag is set to 1 and written into the code stream for transmission to the decoder. The coding unit-level flag ctb_nnlf_flag does not need to be written into the code stream; slice_nnlf_qpadj_flag is written into the code stream for transmission to the decoder. If slice_nnlf_qpadj_flag is true, slice_nnlf_qpadj_index needs to be written into the code stream for transmission to the decoder. In addition, if the current image or slice allows the use of the multiple geometric model input technologies proposed in this article, slice_nnlf_geotransform_flag needs to be written into the input code stream for transmission to the decoder. If slice_nnlf_geotransform_flag is true, slice_nnlf_geotransform_index also needs to be written into the code stream for transmission to the decoder.

[0253] If costCtu is minimum, indicating that not all coding tree units in the current image or slice use neural network loop filtering technology, the image-level or slice-level flag slice_nnlf_flag is set to 2 and written into the bitstream for transmission to the decoder. The neural network loop filtering technology usage flag ctb_nnlf_flag for each coding tree unit also needs to be written into the bitstream. The truth of this flag has been described and determined previously. Slice_nnlf_qpadj_flag is written into the bitstream for transmission to the decoder. If slice_nnlf_qpadj_flag is true, slice_nnlf_qpadj_index is also written into the bitstream for transmission to the decoder. In addition, if the current image or slice allows the use of multiple geometric model input technologies proposed in this article, slice_nnlf_geotransform_flag is written into the input bitstream for transmission to the decoder. If slice_nnlf_geotransform_flag is true, slice_nnlf_geotransform_index is also written into the bitstream for transmission to the decoder.

[0254] 3. The encoder continues to try other loop filtering tools and outputs a complete reconstructed image after completion. The specific process is not relevant to the technology in this article and will not be explained in detail here.

[0255] This technical solution, without adding a new model, uses rate-distortion optimization calculations on the encoder side to determine whether to use input geometric transformation. The decoder side only needs to parse the information in the bitstream, which is beneficial to improving encoding performance. (Negative numbers represent performance gains)

[0256] Table 3

[0257]

[0258] Table 4

[0259]

[0260] Experimental results show that the proposed method improves the encoding performance of all color components under both Random Access and Low Delay B test conditions. Regarding encoding and decoding time, the proposed solution has little impact on decoding time, with only minor fluctuations.

[0261] While the scaling operation is not described in detail in all of the above examples, this does not mean that the present technical solution cannot employ it. The scaling operation is applied to the output of the neural network model, scaling the residual obtained by subtracting the filtered and unfiltered reconstructed samples from the neural network output.

[0262] In the above embodiments, it is not specified whether it is a luminance component or a chrominance component. Since the method proposed in this article is applicable to both luminance components and chrominance components, no limitation is made here. The syntax elements in this article can be clearly seen, such as ctb_nnlf_flag and slice_nnlf_flag, which can be controlled separately for luminance components and chrominance components.

[0263] In yet another embodiment of the present application, the embodiment of the present application further provides a code stream, which is generated by bit encoding according to information to be encoded; wherein the information to be encoded includes at least: relevant syntax elements of the current coding tree unit.

[0264] In yet another embodiment of the present application, referring to FIG11 , a schematic diagram of the structure of a decoding device provided in an embodiment of the present application is shown. As shown in FIG11 , the decoding device may include:

[0265] The decoding unit 1101 is configured to decode the code stream and determine the relevant syntax elements of the current coding tree unit;

[0266] A first determining unit 1102 is configured to determine a geometric transform type of a current coding tree unit according to the relevant syntax elements;

[0267] The second determining unit 1103 is configured to determine reference sample information of the current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit;

[0268] a geometric transformation unit 1104 configured to perform a geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type to obtain geometrically transformed reference sample information;

[0269] The filtering unit 1105 is configured to input the reference sample information after the geometric transformation of the current coding tree unit into the loop filtering model based on the neural network for filtering, and output the filtered reconstructed sample information;

[0270] The geometric inverse transform unit 1106 is configured to perform a geometric inverse transform on the filtered reconstructed sample information according to the geometric transform type to obtain final reconstructed sample information of the current coding tree unit.

[0271] In practical applications, an embodiment of the present application further provides a decoding device. FIG12 is a schematic diagram of a specific hardware structure of a decoding device provided in an embodiment of the present application. As shown in FIG12 , the decoding device includes:

[0272] The first memory 1201 and the first processor 1202; the first memory 1201 stores a computer program that can be run on the first processor 1202, and the first processor 1202 executes the program when the decoding method on the decoder side is executed.

[0273] In practical applications, the decoding device may further include: a first communication interface, used for sending and receiving signals during the process of sending and receiving information with other external network elements.

[0274] In another embodiment of the present application, referring to FIG13 , a schematic diagram of the composition structure of an encoding device provided in an embodiment of the present application is shown. As shown in FIG13 , the encoding device may include:

[0275] The first determining unit 1301 is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and reconstructed sample information of the current coding tree unit;

[0276] The geometric transformation unit 1302 is configured to perform a geometric transformation on the reference sample information of the current coding tree unit according to the candidate geometric transformation type to obtain the reference sample information after the geometric transformation;

[0277] The filtering unit 1303 is configured to input the reference sample information after the geometric transformation of the current coding tree unit into the loop filtering model based on the neural network for filtering, and output the filtered reconstructed sample information;

[0278] The inverse geometric transform unit 1304 is configured to perform an inverse geometric transform on the filtered reconstructed sample information according to the candidate geometric transform type to obtain final reconstructed sample information of the current coding tree unit;

[0279] The second determining unit 1305 is configured to determine a first distortion cost value of the current coding tree unit based on the original sample information of the current coding tree unit and the final reconstructed sample information; and determine a geometric transformation type of the current coding tree unit based on the first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type;

[0280] The coding unit 1306 is configured to encode syntax elements related to the geometric transformation type of the current coding tree unit, and write the obtained coded bits into the bitstream.

[0281] In practical applications, an embodiment of the present application further provides an encoding device. FIG14 is a schematic diagram of a specific hardware structure of an encoding device provided in an embodiment of the present application. As shown in FIG14 , the encoding device includes:

[0282] The second memory 1401 and the second processor 1402; the second memory 1401 stores a computer program that can be run on the second processor 1402, and the second processor 1402 executes the program when the encoding method on the encoder side is executed.

[0283] In practical applications, the encoding device may further include: a second communication interface for receiving and sending signals during the process of sending and receiving information with other external network elements.

[0284] By using the above-mentioned device or equipment, the input information is geometrically transformed and then input into the model for inference calculation, and the output information is inversely transformed to obtain reconstructed sample information, so as to give full play to the optimal performance of the model and have a better filtering effect than direct input.

[0285] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0286] In yet another embodiment of the present application, see Figure 15 , which shows a schematic diagram of the structure of a coding and decoding system provided in an embodiment of the present application. As shown in Figure 15 , the coding and decoding system 150 may include an encoder 1501 and a decoder 1502. The encoder 1501 may be a device integrated with the encoding device described in the previous embodiment, or may be the encoding device described in the previous embodiment; the decoder 1502 may be a device integrated with the decoding device described in the previous embodiment, or may be the decoding device described in the previous embodiment.

[0287] In an embodiment of the present application, in the encoding and decoding system 150, both the encoder 1501 and the decoder 1502 can perform geometric transformation on the input information and input it into the model for inference calculation, and perform inverse transformation on the output information to obtain reconstructed sample information, thereby giving full play to the optimal performance of the model and having a better filtering effect than direct input.

[0288] Accordingly, an embodiment of the present invention further provides a computer storage medium storing a computer program, which, when executed by a first processor, implements a decoding method of a decoder; or, when executed by a second processor, implements an encoding method of an encoder.

[0289] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0290] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0291] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments. The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. Industrial Applicability

[0292] The present application provides a coding and decoding method, apparatus, coding device, decoding device, and storage medium, wherein the decoding method includes: decoding a code stream to determine relevant syntax elements of a current coding tree unit; determining a geometric transformation type of the current coding tree unit based on the relevant syntax elements; performing a geometric transformation on the reference sample information of the current coding tree unit based on the geometric transformation type to obtain reference sample information after the geometric transformation; inputting the reference sample information after the geometric transformation of the current coding tree unit into a loop filtering model based on a neural network for filtering, and outputting filtered reconstructed sample information; performing a geometric inverse transformation on the filtered reconstructed sample information based on the geometric transformation type to obtain the final reconstructed sample information of the current coding tree unit. By geometrically transforming the input information and then inputting it into the model for inference calculation, and performing an inverse transformation on the output information to obtain the reconstructed sample information, the optimal performance of the model is fully utilized, and a better filtering effect is achieved compared to direct input.

Claims

1. A decoding method, the method comprising: Decode the code stream and determine the relevant syntax elements of the current coding tree unit; Determining a geometric transformation type of a current coding tree unit according to the relevant syntax elements; Determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit; Performing a geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type to obtain reference sample information after the geometric transformation; Input the reference sample information after the geometric transformation of the current coding tree unit into the loop filter model based on the neural network for filtering, and output the reconstructed sample information after filtering; Performing an inverse geometric transformation on the filtered reconstructed sample information according to the geometric transformation type to obtain final reconstructed sample information of the current coding tree unit.

2. The method according to claim 1, wherein: include: According to the first syntax element, it is determined whether a current image block where a current coding tree unit is located uses a neural network-based loop filtering technology for performing geometric transformation on an input.

3. The method according to claim 2, wherein: The first syntax element includes at least one of the following: The first syntax element at the image sequence level is used to indicate whether the image sequence uses a neural network-based loop filtering technique for geometric transformation of the input; The first syntax element at the image level is used to indicate whether the image uses a neural network-based loop filtering technique for geometric transformation of the input; The first syntax element at the slice level is used to indicate whether the slice uses a neural network-based loop filtering technique for geometric transformation of the input; The first syntax element at the coding tree unit level is used to indicate whether the coding tree unit uses a neural network-based loop filtering technique for geometric transformation of the input.

4. The method according to claim 2, wherein: include: When determining, according to the first syntax element, that a current image block where a current coding tree unit is located uses a neural network-based loop filtering technique for geometric transformation of an input, determining, according to the second syntax element, a geometric transformation type of the coding tree unit in the current image block.

5. The method according to claim 4, wherein: The second syntax element includes one of the following: The second syntax element at the picture sequence level is used to indicate the geometric transformation type of all coding tree units in the picture sequence; The second syntax element at the picture level is used to indicate the geometric transformation type of all coding tree units in the picture; The second syntax element at the slice level is used to indicate the geometric transform type of all coding tree units in the slice; The second syntax element at the coding tree unit level is used to indicate the geometric transform type of the coding tree unit.

6. The method according to any one of claims 2 to 5, wherein: include: According to the third syntax element, it is determined whether the current image block uses a neural network-based loop filtering technology.

7. The method according to claim 6, wherein: The third syntax element includes at least one of the following: The third syntax element at the image sequence level is used to indicate whether the image sequence uses a neural network-based loop filtering technique; The third syntax element at the image level is used to indicate whether the image uses a neural network-based loop filtering technique; The third syntax element at the slice level is used to indicate whether the slice uses a neural network-based loop filtering technique; The third syntax element at the coding tree unit level is used to indicate whether the coding tree unit uses a neural network-based loop filtering technique.

8. The method according to any one of claims 2 to 7, wherein: include: According to the fourth syntax element, it is determined whether the current image block allows the use of a neural network-based loop filtering technique for geometric transformation of the input.

9. The method according to claim 8, wherein: The fourth syntax element includes at least one of the following: A fourth syntax element at the image sequence level, used to indicate whether the image sequence allows the use of a neural network-based loop filtering technique with geometric transformation of the input; The fourth syntax element at the image level is used to indicate whether the image allows the use of neural network-based loop filtering techniques for geometric transformation of the input; The fourth syntax element at the slice level is used to indicate whether the slice allows the use of neural network-based loop filtering techniques for geometric transformation of inputs; The fourth syntax element at the coding tree unit level is used to indicate whether the coding tree unit allows the use of neural network-based loop filtering techniques for geometric transformation of the input.

10. The method according to any one of claims 2 to 9, wherein: include: According to the fifth syntax element, it is determined whether the current image block allows the use of a neural network-based loop filtering technique.

11. The method according to claim 10, wherein: The fifth syntax element includes at least one of the following: The fifth syntax element at the image sequence level is used to indicate whether the image sequence allows the use of neural network-based loop filtering technology; The fifth syntax element at the image level is used to indicate whether the image allows the use of neural network-based loop filtering technology; The fifth syntax element at the slice level is used to indicate whether the slice allows the use of neural network-based loop filtering technology; The fifth syntax element at the coding tree unit level is used to indicate whether the coding tree unit is allowed to use the neural network based loop filtering technology.

12. The method according to any one of claims 1 to 11, wherein: The parameter sample information also includes: constant parameters; the method also includes: Determining whether to adjust constant parameters of a current image block where a current coding tree unit is located according to a sixth syntax element; When determining, according to the sixth syntax element, to adjust the constant parameters of the current image block where the current coding tree unit is located, determining, according to the seventh syntax element, the adjusted constant parameters of the current image block where the current coding tree unit is located; The constant parameters are adjusted according to the adjustment parameters, and the adjusted constant parameters are input into the neural network-based loop filtering technology.

13. The method according to claim 12, wherein: The sixth syntax element includes at least one of the following: The sixth syntax element at the picture sequence level is used to indicate whether to adjust the constant parameters of the coding tree units in the picture sequence; The sixth syntax element at the picture level is used to indicate whether to adjust the constant parameters of the coding tree units in the picture; The sixth syntax element at the slice level is used to indicate whether to adjust the constant parameters of the coding tree units in the slice; The sixth syntax element at the coding tree unit level is used to indicate whether to adjust the constant parameters of the coding tree unit; The seventh syntax element includes one of the following: The seventh syntax element at the picture sequence level is used to indicate the adjusted constant parameters of all coding tree units in the picture sequence; The seventh syntax element at the picture level is used to indicate the adjusted constant parameters of all coding tree units in the picture; The seventh syntax element at the slice level is used to indicate the adjusted constant parameters of all coding tree units in the slice; The seventh syntax element at the coding tree unit level is used to indicate the adjusted constant parameters of the coding tree unit.

14. The method according to any one of claims 2 to 13, wherein: The current image block includes at least one of the following: a current image sequence, a current image, a current slice, and a current coding tree unit.

15. The method according to any one of claims 1 to 14, wherein: include: When the time domain level of the current coding tree unit is greater than or equal to the time domain level threshold, determining the geometric transformation type of the current coding tree unit according to the relevant syntax element; When the time domain level of the current coding tree unit is less than the time domain level threshold, it is determined not to use the neural network-based loop filtering technology for geometric transformation of the input.

16. The method according to any one of claims 1 to 15, wherein: The reference sample information also includes constant parameters and non-constant parameters of the current tree coding unit; The constant parameter includes at least one of the following: a quantization parameter, an image type or a slice type corresponding to the current coding tree unit; The non-constant parameter includes at least one of the following: boundary strength information of the current coding tree unit, division information of the current coding tree unit, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image.

17. The method according to any one of claims 1 to 15, wherein: The geometric transformation type includes one of the following: diagonal flip, horizontal flip, vertical flip, and rotation by a preset angle.

18. The method according to any one of claims 1 to 15, wherein: The current coding tree unit is a maximum coding unit, or is obtained by scaling the maximum coding unit.

19. A coding method, the method comprising: Determine reference sample information of a current coding tree unit; wherein the reference sample information includes at least: predicted sample information and / or reconstructed sample information of the current coding tree unit; Performing geometric transformation on the reference sample information of the current coding tree unit according to the candidate geometric transformation type to obtain reference sample information after geometric transformation; Input the reference sample information after the geometric transformation of the current coding tree unit into the loop filter model based on the neural network for filtering, and output the reconstructed sample information after filtering; Performing an inverse geometric transformation on the filtered reconstructed sample information according to the candidate geometric transformation type to obtain final reconstructed sample information of the current coding tree unit; Determine a first distortion cost value of the current coding tree unit according to original sample information of the current coding tree unit and the final reconstructed sample information; Determining a geometric transformation type of the current coding tree unit according to a first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type; Encode the relevant syntax elements of the geometric transformation type of the current coding tree unit, and write the obtained coded bits into the bitstream.

20. The method according to claim 19, wherein: include: The first syntax element is set according to whether a current image block where a current coding tree unit is located uses a neural network-based loop filtering technology for performing a geometric transformation using an input.

21. The method according to claim 20, wherein: The first syntax element includes at least one of the following: The first syntax element at the image sequence level is used to indicate whether the image sequence uses a neural network-based loop filtering technique for geometric transformation of the input; The first syntax element at the image level is used to indicate whether the image uses a neural network-based loop filtering technique for geometric transformation of the input; The first syntax element at the slice level is used to indicate whether the slice uses a neural network-based loop filtering technique for geometric transformation of the input; The first syntax element at the coding tree unit level is used to indicate whether the coding tree unit uses a neural network-based loop filtering technique for geometric transformation of the input.

22. The method according to claim 20, wherein: include: When determining that a current image block where a current coding tree unit is located uses a neural network-based loop filtering technique for geometric transformation of an input, a second syntax element is set according to a geometric transformation type of the coding tree unit in the current image block.

23. The method according to claim 22, wherein: The second syntax element includes one of the following: The second syntax element at the picture sequence level is used to indicate the geometric transformation type of all coding tree units in the picture sequence; The second syntax element at the picture level is used to indicate the geometric transformation type of all coding tree units in the picture; The second syntax element at the slice level is used to indicate the geometric transform type of all coding tree units in the slice; The second syntax element at the coding tree unit level is used to indicate the geometric transform type of the coding tree unit.

24. The method according to any one of claims 20 to 23, wherein: include: The third syntax element is set according to whether the current image block uses a neural network-based loop filtering technology.

25. The method according to claim 24, wherein: The third syntax element includes at least one of the following: The third syntax element at the image sequence level is used to indicate whether the image sequence uses a neural network-based loop filtering technique; The third syntax element at the image level is used to indicate whether the image uses a neural network-based loop filtering technique; The third syntax element at the slice level is used to indicate whether the slice uses a neural network-based loop filtering technique; The third syntax element at the coding tree unit level is used to indicate whether the coding tree unit uses a neural network-based loop filtering technique.

26. The method according to any one of claims 20 to 25, wherein: include: The fourth syntax element is set according to whether the current image block allows the use of a neural network-based loop filtering technique for geometric transformation of the input.

27. The method according to claim 26, wherein: The fourth syntax element includes at least one of the following: A fourth syntax element at the image sequence level, used to indicate whether the image sequence allows the use of a neural network-based loop filtering technique with geometric transformation of the input; The fourth syntax element at the image level is used to indicate whether the image allows the use of neural network-based loop filtering techniques for geometric transformation of the input; The fourth syntax element at the slice level is used to indicate whether the slice allows the use of neural network-based loop filtering techniques for geometric transformation of inputs; The fourth syntax element at the coding tree unit level is used to indicate whether the coding tree unit allows the use of neural network-based loop filtering techniques for geometric transformation of the input.

28. The method according to any one of claims 20 to 27, wherein: include: The fifth syntax element is set according to whether the current image block allows the use of a neural network-based loop filtering technique.

29. The method according to claim 28, wherein: The fifth syntax element includes at least one of the following: The fifth syntax element at the image sequence level is used to indicate whether the image sequence allows the use of neural network-based loop filtering technology; The fifth syntax element at the image level is used to indicate whether the image allows the use of neural network-based loop filtering technology; The fifth syntax element at the slice level is used to indicate whether the slice allows the use of neural network-based loop filtering technology; The fifth syntax element at the coding tree unit level is used to indicate whether the coding tree unit is allowed to use the neural network based loop filtering technology.

30. The method according to any one of claims 19 to 29, wherein: The determining, according to the first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type, the geometric transformation type of the current coding tree unit includes: Accumulating the first distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the first distortion cost value of the current image block; Determine the geometric transformation type of the current coding tree unit according to the first distortion cost value of the current image block corresponding to the candidate geometric transformation type.

31. The method according to claim 30, wherein: Also includes: Input the reference sample information of the current coding tree unit into the loop filter model based on the neural network for filtering, and output the filtered reconstructed sample information; Determine a second distortion cost value of the current coding tree unit according to the original sample information of the current coding tree unit and the filtered reconstructed sample information; Accumulating the second distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the second distortion cost value of the current image block; Determine a third distortion cost value of the current coding tree unit according to original sample information and reconstructed sample information of the current coding tree unit; Accumulating the third distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the third distortion cost value of the current image block; Using the minimum distortion cost value among the first distortion cost value, the second distortion cost value and the third distortion cost value of the current coding tree unit as the fourth distortion cost value of the current coding tree unit; Accumulating the fourth distortion cost values ​​of all coding tree units in the current image block where the current coding tree unit is located to determine the fourth distortion cost value of the current image block; According to the minimum distortion cost value among the first distortion cost value, the second distortion cost value, the third distortion cost value and the fourth distortion cost value of the current image block, it is determined whether a neural network-based loop filtering technology for input geometric transformation is used in the current image block, and whether a neural network-based loop filtering technology is used in the current image block.

32. The method according to any one of claims 19 to 31, wherein: The reference sample information also includes: candidate constant parameters of the current coding tree unit; The method further includes: determining a target constant parameter of the current image block where the current coding tree unit is located according to a first distortion cost value of the current image block corresponding to the candidate geometric transformation type.

33. The method of claim 32, wherein: The method further comprises: According to the target constant parameters of the current image block where the current coding tree unit is located, determining whether to adjust the constant parameters of the current image block where the current coding tree unit is located, and setting the sixth syntax element; It is determined to adjust constant parameters of the current image block where the current coding tree unit is located, and a seventh syntax element is set according to target constant parameters of the current image block where the current coding tree unit is located.

34. The method of claim 33, wherein: The sixth syntax element includes at least one of the following: The sixth syntax element at the picture sequence level is used to indicate whether to adjust the constant parameters of the coding tree units in the picture sequence; The sixth syntax element at the picture level is used to indicate whether to adjust the constant parameters of the coding tree units in the picture; The sixth syntax element at the slice level is used to indicate whether to adjust the constant parameters of the coding tree units in the slice; The sixth syntax element at the coding tree unit level is used to indicate whether to adjust the constant parameters of the coding tree unit; The seventh syntax element includes one of the following: The seventh syntax element at the picture sequence level is used to indicate the adjusted constant parameters of all coding tree units in the picture sequence; The seventh syntax element at the picture level is used to indicate the adjusted constant parameters of all coding tree units in the picture; The seventh syntax element at the slice level is used to indicate the adjusted constant parameters of all coding tree units in the slice; The seventh syntax element at the coding tree unit level is used to indicate the adjusted constant parameters of the coding tree unit.

35. The method according to any one of claims 20 to 34, wherein: The current image block includes at least one of the following: a current image sequence, a current image, a current slice, and a current coding tree unit.

36. The method according to any one of claims 19 to 35, wherein: include: When the temporal level of the current coding tree unit is greater than or equal to the temporal level threshold, determining the geometric transformation type of the current coding tree unit according to the first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type; When the time domain level of the current coding tree unit is less than the time domain level threshold, it is determined not to use the neural network-based loop filtering technology for geometric transformation of the input.

37. The method according to any one of claims 19 to 35, wherein: The reference sample information also includes constant parameters and non-constant parameters of the current tree coding unit; The constant parameter includes at least one of the following: a quantization parameter, an image type or a slice type corresponding to the current coding tree unit; The non-constant parameter includes at least one of the following: boundary strength information of the current coding tree unit, division information of the current coding tree unit, and reconstructed sample information of the coding tree unit corresponding to the current coding tree unit in the reference image.

38. The method according to any one of claims 19 to 35, wherein: The geometric transformation type includes one of the following: diagonal flip, horizontal flip, vertical flip, and rotation by a preset angle.

39. The method according to any one of claims 19 to 35, wherein: The current coding tree unit is a maximum coding unit, or is obtained by scaling the maximum coding unit.

40. A decoding device, wherein: The decoding device comprises: A decoding unit configured to decode the bitstream and determine relevant syntax elements of a current coding tree unit; A first determining unit, configured to determine a geometric transformation type of a current coding tree unit according to the relevant syntax element; A second determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information at least includes: predicted sample information and / or reconstructed sample information of the current coding tree unit; a geometric transformation unit, configured to perform a geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type to obtain the reference sample information after the geometric transformation; A filtering unit, configured to input the reference sample information after the geometric transformation of the current coding tree unit into a loop filtering model based on a neural network for filtering, and output the filtered reconstructed sample information; The geometric inverse transformation unit is configured to perform a geometric inverse transformation on the filtered reconstructed sample information according to the geometric transformation type to obtain the final reconstructed sample information of the current coding tree unit.

41. An encoding device, wherein: The encoding device comprises: A first determining unit is configured to determine reference sample information of a current coding tree unit; wherein the reference sample information at least includes: predicted sample information and reconstructed sample information of the current coding tree unit; A geometric transformation unit, configured to perform a geometric transformation on the reference sample information of the current coding tree unit according to the candidate geometric transformation type to obtain the reference sample information after the geometric transformation; A filtering unit configured to input the reference sample information after the geometric transformation of the current coding tree unit into a loop filtering model based on a neural network for filtering, and output the reconstructed sample information after filtering; a geometric inverse transformation unit configured to perform a geometric inverse transformation on the filtered reconstructed sample information according to the candidate geometric transformation type to obtain final reconstructed sample information of the current coding tree unit; a second determining unit configured to determine a first distortion cost value of the current coding tree unit according to the original sample information of the current coding tree unit and the final reconstructed sample information; and determine a geometric transformation type of the current coding tree unit according to the first distortion cost value of the current coding tree unit corresponding to the candidate geometric transformation type; The coding unit is configured to encode the relevant syntax elements of the geometric transformation type of the current coding tree unit, and write the obtained coded bits into the bitstream.

42. A decoding device, wherein: The decoding device comprises: a first memory and a first processor; The first memory stores a computer program that can be run on the first processor, and the first processor implements the decoding method according to any one of claims 1 to 18 when executing the program.

43. An encoding device, wherein: The encoding device includes: a second memory and a second processor; The second memory stores a computer program that can be run on the second processor, and the second processor implements the encoding method according to any one of claims 19 to 39 when executing the program.

44. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a first processor, it implements the decoding method described in any one of claims 1 to 18; or, when the computer program is executed by a second processor, it implements the encoding method described in any one of claims 19 to 39.

Images