Loop filtering method, video encoding and decoding method, device and system

Abstract

The invention discloses a loop filtering method, a video coding and decoding method, a device and a system, and the method comprises the steps: inputting a reconstructed image into a neural network for region-based loop filtering in a video coding and decoding process, i.e., dividing the reconstructed image into one or more first regions, determining filtering parameters based on the first regions, and carrying out the loop filtering. When loop filtering based on the residual neural network is carried out, one scaling factor selected from N candidate scaling factors can be used for scaling the global residual of the residual neural network, and N is greater than or equal to 1. The embodiment of the invention further provides a corresponding device and system. According to the embodiment of the invention, the flexibility of video coding and decoding can be improved, and the coding performance is improved.

Description

A loop filtering method, video encoding and decoding method, device and system Technical Field

[0001] The embodiments of the present disclosure relate to, but are not limited to, video technology, and more specifically, to a loop filtering method, a video encoding and decoding method, a device, and a system. Background Art

[0002] Adaptive loop filtering (ALF) is a technology used in video codec standards. It uses adaptive filtering parameters to improve the objective and subjective quality of each reconstructed image compared to the original. ALF filter parameters are obtained by the encoder and written into the bitstream for use by the decoder, but filtering effectiveness still needs to be improved.

[0003] SUMMARY OF THE INVENTION

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] An embodiment of the present disclosure provides a loop filtering method, including:

[0006] The reconstructed image is input into the neural network for region-based loop filtering;

[0007] Obtaining a reconstructed image after loop filtering from the output of the neural network;

[0008] The region-based loop filtering is to divide the reconstructed image into one or more first regions, determine filtering parameters based on the first regions and perform loop filtering, wherein the first regions include one or more coding tree units.

[0009] An embodiment of the present disclosure further provides a loop filtering method, including:

[0010] The reconstructed image is input into the residual neural network for loop filtering;

[0011] Scaling the global residual of the residual neural network using a scaling factor, wherein the scaling factor is selected from N candidate scaling factors, where N is a variable greater than or equal to 1 or a fixed value greater than 1;

[0012] The scaled global residual is added to the reconstructed image input to the residual neural network, and the reconstructed image after loop filtering is output.

[0013] An embodiment of the present disclosure further provides a video decoding method, including:

[0014] Decode the code stream to obtain the filtering parameters used when performing loop filtering on the reconstructed image;

[0015] The reconstructed image is subjected to loop filtering using the obtained filtering parameters, wherein the loop filtering is performed according to the loop filtering method described in any embodiment of the present disclosure.

[0016] An embodiment of the present disclosure further provides a video encoding method, including:

[0017] Obtain a reconstructed image based on the predicted image and the residual;

[0018] Perform loop filtering on the reconstructed image to obtain a reconstructed image after loop filtering, and encode filtering parameters used in the loop filtering; wherein the loop filtering is performed according to the loop filtering method described in any embodiment of the present disclosure.

[0019] An embodiment of the present disclosure further provides a loop filtering device, comprising a processor and a memory storing a computer program, wherein the processor can implement the loop filtering method as described in any embodiment of the present disclosure when executing the computer program.

[0020] An embodiment of the present disclosure further provides a video decoding device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, the video decoding method as described in any one of the embodiments of the present disclosure is implemented.

[0021] An embodiment of the present disclosure further provides a video encoding device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, the video encoding method as described in any embodiment of the present disclosure is implemented.

[0022] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes the video encoding device as described in any embodiment of the present disclosure and the video decoding device as described in any embodiment of the present disclosure.

[0023] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program implements the method described in any embodiment of the present disclosure when executed by a processor.

[0024] An embodiment of the present disclosure further provides a code stream, wherein the code stream is generated by the video encoding method as described in any embodiment of the present disclosure.

[0025] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0026] Summary of the Figures

[0027] The accompanying drawings are used to provide an understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation to the technical solutions of the present disclosure.

[0028] FIG1A is a schematic diagram of a coding and decoding system according to an embodiment of the present disclosure;

[0029] FIG1B is a framework diagram of an encoding end according to an embodiment of the present disclosure;

[0030] FIG1C is a framework diagram of a decoding end according to an embodiment of the present disclosure;

[0031] FIG2 is a flow chart of a loop filtering method according to an embodiment of the present disclosure;

[0032] FIG3A is a structural diagram of a CNNLF network composed of three convolutional layers according to an embodiment of the present disclosure;

[0033] FIG3B is a structural diagram of a CNNLF network in which the global residual is scaled based on FIG3A ;

[0034] FIG4A is a structural diagram of a CNNLF network composed of multiple residual blocks according to an embodiment of the present disclosure;

[0035] FIG4B is a structural diagram of a CNNLF network in which the global residual is scaled based on FIG4A ;

[0036] FIG5 is an exemplary structural diagram of the residual block in FIG4A ;

[0037] FIG6 is a schematic diagram of a plurality of second regions into which an image is divided and their indexes according to an embodiment of the present disclosure;

[0038] FIG7 is a schematic diagram of merging the second area in FIG6 into the first area according to an embodiment of the present disclosure;

[0039] FIG8 is a schematic diagram of performing multi-branch tree partitioning on an image in a recursive manner according to an embodiment of the present disclosure;

[0040] FIG9 is a flow chart of another loop filtering method according to an embodiment of the present disclosure;

[0041] FIG10 is a flowchart of a video encoding method according to an embodiment of the present disclosure;

[0042] FIG11 is a flowchart of a video decoding method according to an embodiment of the present disclosure;

[0043] FIG12 is a schematic diagram of a loop filtering device according to an embodiment of the present disclosure.

[0044] Details

[0045] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure.

[0046] In the description of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment described as "exemplary" or "for example" in the present disclosure should not be interpreted as being more preferred or advantageous than other embodiments. "And / or" in this article is a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "Multiple" refers to two or more than two. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0047] When describing representative exemplary embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can readily understand that these sequences can vary and still remain within the spirit and scope of the disclosed embodiments.

[0048] The loop filtering method proposed in the embodiments of the present disclosure can be applied to various video coding and decoding standards, such as: H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), AVS (Audio Video Coding Standard), and other standards developed by MPEG (Moving Picture Experts Group), AOM (Alliance for Open Media), JVET (Joint Video Experts Team) and extensions of these standards, or any other customized standards.

[0049] Figure 1A is a block diagram of a video encoding and decoding system that can be used in embodiments of the present disclosure. As shown, the system comprises an encoding device 1 and a decoding device 2. Encoding device 1 generates a bitstream. Decoding device 2 decodes the bitstream. Encoding device 1 and decoding device 2 may include one or more processors and memory coupled to the one or more processors.

[0050] The decoding end device 2 can receive the code stream from the encoding end device 1 via the link 3. The link 3 includes one or more media or devices capable of moving the code stream from the encoding end device 1 to the decoding end device 2. In one example, the link 3 includes one or more communication media that enable the encoding end device 1 to send the code stream directly to the decoding end device 2. The encoding end device 1 can modulate the code stream according to a communication standard (such as a wireless communication protocol) and can send the modulated code stream to the decoding end device 2. The one or more communication media may include wireless and / or wired communication media and may form part of a packet network. In another example, the code stream can also be output from the output interface 15 to a storage device, and the decoding end device 2 can read the stored data from the storage device via streaming or downloading.

[0051] In the example shown in FIG1A , the encoding end device 1 includes a data source 11, a video encoding device 13, and an output interface 15. The data source 11 may include a video capture device (e.g., a camera), an archive containing previously captured data, a feed interface for receiving data from a content provider, a computer graphics system for generating data, or a combination of these sources. The video encoding device 13 encodes the data from the data source 11 and outputs it to the output interface 15. The output interface 15 may include at least one of a regulator, a modem, and a transmitter. The decoding end device 2 includes an input interface 21, a video decoding device 23, and a display device 25. The input interface 21 includes at least one of a receiver and a modem. The input interface 21 may receive a code stream via the link 3 or from a storage device. The video decoding device 23 decodes the received code stream. The display device 25 is used to display the decoded data. The display device 25 may be integrated with other devices of the decoding end device 2 or provided separately. The display device 25 is optional for the decoding end. In other examples, the decoding end may include other devices or equipment that apply the decoded data.

[0052] Based on the video coding and decoding system shown in FIG. 1A , various video coding and decoding methods can be used to implement video compression and decompression.

[0053] FIG1B is a block diagram of an exemplary video encoding device that can be used in embodiments of the present disclosure. As shown, the video encoding device 1000 includes a prediction processing unit 1100, a partitioning unit 1101, a residual generation unit 1102 (indicated by a circle with a plus sign after the partitioning unit in the figure), a transform processing unit 1104, a quantization unit 1106, an inverse quantization unit 1108, an inverse transform processing unit 1110, a reconstruction unit 1112 (indicated by a circle with a plus sign after the inverse transform processing unit in the figure), a filter unit 1113, a decoded picture buffer 1114, an image resolution adjustment unit 1115, and an entropy coding unit 1116. The prediction processing unit 1100 includes an inter-frame prediction processing unit 121 and an intra-frame prediction processing unit 1126. The video encoder 20 may also include more, fewer, or different functional components than shown in this example.

[0054] The division unit 1101 cooperates with the prediction processing unit 1100 to divide the received video data into slices, coding tree units (CTUs), or other larger units. The video data received by the division unit 1101 can be a video sequence including video frames such as I frames, P frames, or B frames.

[0055] The prediction processing unit 1100 can divide a CTU into coding units (CUs) and perform intra-frame prediction or inter-frame prediction on the CU. When performing intra-frame prediction or inter-frame prediction on the CU, the CU can be divided into one or more prediction units (PUs).

[0056] The inter-prediction processing unit 1121 may perform inter prediction on a PU and generate prediction data for the PU, where the prediction data includes a prediction block of the PU, motion information of the PU, and various syntax elements.

[0057] Intra-prediction processing unit 1126 may perform intra prediction on a PU and generate prediction data for the PU, which may include a prediction block and various syntax elements for the PU.

[0058] The residual generation unit 1102 may subtract the prediction block of the PU partitioned by the CU from the original block of the CU to generate a residual block of the CU.

[0059] The transform processing unit 1104 may divide a CU into one or more transform units (TUs). A residual block associated with a TU is a sub-block obtained by dividing the residual block of the CU. One or more transforms are applied to the residual block associated with the TU to generate a coefficient block associated with the TU.

[0060] The quantization unit 1106 may quantize the coefficients in the coefficient block based on the selected quantization parameter. The degree of quantization of the coefficient block may be adjusted by adjusting the quantization parameter (QP: Quantizer Parameter) value.

[0061] The inverse quantization unit 1108 and the inverse transform unit 1110 may apply inverse quantization and inverse transform to the coefficient block, respectively, to obtain a reconstructed residual block associated with the TU.

[0062] The reconstruction unit 1112 may add the reconstructed residual block and the prediction block generated by the prediction processing unit 1100 to generate a reconstructed image.

[0063] Filter unit 1113 performs loop filtering on the reconstructed image and stores the filtered reconstructed image in decoded picture buffer 1114 as a reference image. Intra-prediction processing unit 1126 extracts reference images of blocks adjacent to the PU from decoded picture buffer 1114 to perform intra-prediction. Inter-prediction processing unit 1121 performs inter-prediction on the PU of the current frame using the reference image of the previous frame cached in decoded picture buffer 1114.

[0064] The image resolution adjustment unit 1115 resamples the reference image stored in the decoded image buffer 1114 , which may include upsampling and / or downsampling, to obtain reference images of multiple resolutions and store them in the decoded image buffer 1114 .

[0065] The entropy coding unit 1116 may perform entropy coding operations on received data (eg, syntax elements, quantized coefficient blocks, motion information, etc.).

[0066] FIG1C is a block diagram of an exemplary video decoding device that can be used in embodiments of the present disclosure. As shown, the video decoding device 101 includes an entropy decoding unit 150, a prediction processing unit 152, an inverse quantization unit 154, an inverse transform processing unit 156, a reconstruction unit 158 ​​(represented by a circle with a plus sign after the inverse transform processing unit), a filter unit 159, and a picture buffer 160. In other embodiments, the video decoder 30 may include more, fewer, or different functional components.

[0067] Entropy decoding unit 150 may perform entropy decoding on the received bitstream to extract syntax elements, quantized coefficient blocks, motion information for the PU, etc. Prediction processing unit 152, inverse quantization unit 154, inverse transform processing unit 156, reconstruction unit 158, and filter unit 159 may each perform corresponding operations based on the syntax elements extracted from the bitstream.

[0068] As a functional component performing reconstruction operations, inverse quantization unit 154 may inverse quantize the coefficient block associated with the quantized TU. Inverse transform processing unit 156 may apply one or more inverse transforms to the inverse quantized coefficient block to generate a reconstructed residual block for the TU.

[0069] Prediction processing unit 152 includes inter-prediction processing unit 162 and intra-prediction processing unit 164. If a PU is encoded using intra prediction, intra-prediction processing unit 164 may determine an intra prediction mode for the PU based on syntax elements parsed from the codestream, perform intra prediction based on the determined intra prediction mode and reconstructed reference information of the PU's neighbors obtained from picture buffer 60, and generate a prediction block for the PU. If a PU is encoded using inter prediction, inter-prediction processing unit 162 may determine one or more reference blocks for the PU based on the motion information of the PU and corresponding syntax elements, and generate a prediction block for the PU based on the reference blocks.

[0070] The reconstruction unit 158 ​​may obtain a reconstructed image based on the reconstructed residual block associated with the TU and the prediction block of the PU generated by the prediction processing unit 152 (ie, intra-frame prediction data or inter-frame prediction data).

[0071] The filter unit 159 may perform loop filtering on the reconstructed image, and the filtered reconstructed image is stored in the picture buffer 160. The picture buffer 160 may provide a reference picture for subsequent motion compensation, intra-frame prediction, inter-frame prediction, etc., and may also output the filtered reconstructed image as decoded video data for presentation on a display device.

[0072] The display 105 may be a liquid crystal display, a plasma display, an organic light emitting diode display, or other types of display devices.

[0073] Based on the above-described video encoding and decoding devices, at the encoding end, a frame of an image is divided into blocks. Intra-frame prediction, inter-frame prediction, or other algorithms are used on the current block to generate a prediction block for the current block. The prediction block is subtracted from the original block of the current block to obtain a residual block. The residual block is transformed and quantized to obtain quantized coefficients. These quantized coefficients are then entropy encoded to generate a bitstream. At the decoding end, intra-frame prediction or inter-frame prediction is used on the current block to generate a prediction block for the current block. The quantized coefficients obtained by parsing the bitstream are then inversely quantized and inversely transformed to obtain a residual block. The prediction block and residual block are added together to obtain a reconstructed block. The reconstructed image is then subjected to image-based or block-based loop filtering. The resulting loop-filtered reconstructed image can be used as the decoded image. The encoding end also requires similar operations as the decoding end to obtain the decoded image. The decoded image can be used as a reference image for inter-frame prediction. The block division information determined by the encoding end, as well as information or parameters related to prediction, transform, quantization, entropy coding, and loop filtering modes, can be written into the bitstream as syntax elements. The decoding end determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information as the encoding end by parsing or analyzing the existing information, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end.

[0074] Although the above is an example of a block-based hybrid coding framework, the embodiments of the present disclosure are not limited to this. Some modules or steps of the framework or process can be replaced, for example, by other implementation methods such as a neural network-based model to obtain a reconstructed image.

[0075] With the continuous development of neural networks, neural network-based video coding (NNVC) technology has become a research direction for improving video coding efficiency. In one embodiment, a convolutional neural network-based loop filter (CNNLF) is used. CNNLF uses more filtering parameters and higher computational complexity, resulting in further improved filtering performance compared to ALF.

[0076] An embodiment of the present disclosure provides a loop filtering method, as shown in FIG2 , including:

[0077] Step 110, inputting the reconstructed image into a neural network for region-based loop filtering;

[0078] Step 120: Obtain a reconstructed image after loop filtering from the output of the neural network.

[0079] The region-based loop filtering is to divide the reconstructed image into one or more first regions, determine filtering parameters based on the first regions and perform loop filtering, wherein the first regions include one or more coding tree units.

[0080] The first region used in the region-based loop filtering of this embodiment is a filtering unit between an image and a coding tree unit. At least one of the first regions into which an image is divided includes multiple coding tree units.

[0081] The region-based loop filtering of this embodiment is different from image-based loop filtering and block-based loop filtering. Image-based filtering determines filtering parameters based on an image (picture) and performs loop filtering. The image here can be a video frame or a video frame divided into slices, slice segments, tiles, and other parallel-encoded coding units. Block-based filtering determines filtering parameters based on coding tree blocks (CTBs) and performs loop filtering. When using a neural network as a filter, image-based loop filtering is not flexible enough and cannot select better filtering parameters based on the differences between different image regions. Block-based loop filtering, on the other hand, has high computational and transmission overhead. The region-based loop filtering of this embodiment determines filtering parameters based on the first region (i.e., filtering parameters are determined for each first region separately, and loop filtering is performed on that region using the filtering parameters of that region). This can achieve a better balance between image quality and overhead, thereby improving overall encoding performance.

[0082] The neural network shown in Figure 3A can be used to implement the loop filtering method of this embodiment. The neural network is a network model of CNNLF, which can be used after the ALF stage of loop filtering or can be used independently. The neural network uses a three-layer convolutional network L1, L2 and L3 as the basic structure, with a packing layer before the three-layer convolutional network and an unpacking layer after the three-layer convolutional network. In the example shown in the figure, the color component input to the neural network is in YUV format, and the luminance component Y and the two chrominance components U and V share a set of network parameters for every 32 frames. The parameters of each layer of the convolutional network are M×N×P×Q, where M is the horizontal width of the channel filter, N is the vertical height of the channel filter, P is the number of input channels, and Q is the number of filters in the network layer. The parameters of each layer of the convolutional network in the figure are 3×3×6×8, but this is only exemplary. The neural network uses residual connections globally and is a residual neural network. In this paper, the output of the last layer of the residual neural network is called the global residual of the residual neural network (the output of the Unpacking layer in Figure 3A). The global residual is added to the input of the residual neural network to obtain the output of the residual neural network.

[0083] Based on the CNNLF network model, the Packing layer generates the input for the Convolutional Neural Network (CNN). After passing through three CNN layers and the Unpacking layer, the output is a reconstructed image after loop filtering. In one example, each convolutional network layer performs operations such as convolution (Conv), batch normalization (Batch Normalization), bias addition, and activation (ReLU), but the last convolutional network layer does not need to perform an activation operation.

[0084] The residual neural network shown in Figure 3A can be trained online or pre-trained. In the case of pre-training, the network parameters can be optimized using the gradient descent method to obtain a trained CNNLF network model. Since the network parameters are obtained through a common training set, the network generalization is better. However, a single pre-trained residual neural network is difficult to adapt to different coding parameters (such as image resolution, quantization parameters, etc.). Although multiple residual neural networks with different pre-training parameters can be used as candidates to obtain improved coding performance, the complexity will increase significantly.

[0085] In order to simplify the pre-trained residual neural network from both software and hardware aspects, a scaling factor α can be used to adjust the global residual of the residual neural network to improve the adaptability of the residual neural network to different encoding parameters. The improved residual neural network is shown in Figure 3B, which is basically the same as the residual neural network shown in Figure 3A, except that the global residual is multiplied by the scaling factor α and then added to the input of the neural network to obtain a reconstructed image after loop filtering. The residual neural network shown in Figure 3B can also be used to implement the loop filtering method of this embodiment.

[0086] Another exemplary neural network that can be used to implement the loop filtering method of this embodiment is shown in Figure 4A. This neural network is also a network model of CNNLF, including multiple residual blocks (residual blocks) (represented as ResBlock in the figure. The residual blocks in this neural network are different from the residual blocks generated by subtracting the prediction blocks from the original blocks during video encoding). Each residual block contains at least one convolutional layer (Conv) and at least one activation function (Activation). Figure 5 shows an exemplary structure of a residual block, including two convolutional layers and an activation function located between the two convolutional layers, but this embodiment is not limited to this. The neural network shown in Figure 4A also uses residual connections as a whole. The output of the last residual block is added to the input of the neural network to obtain the output of the neural network. This neural network is also a residual neural network, and the global residual of the residual neural network is the output of the last residual block. Similarly, the residual neural network shown in Figure 4A can be improved to obtain the residual neural network shown in Figure 4B. Based on Figure 4A, this residual neural network multiplies the global residual by a scaling factor α to scale it, and then adds it to the input of the residual neural network to obtain a reconstructed image after loop filtering.

[0087] In one example of this embodiment, when the neural network used to implement loop filtering is a residual neural network and a scaling factor is used to scale the global residual, the filtering parameters include the scaling factor. The region-based loop filtering of this embodiment includes: for each first region, inputting the reconstructed image of the first region into the residual neural network, scaling the global residual of the residual neural network using a scaling factor selected for the first region, adding the scaled global residual to the reconstructed image of the first region input into the residual neural network, and obtaining the reconstructed image of the first region after loop filtering. The reconstructed image of the same first region can be input into the residual neural network at one time, or can be input into the residual neural network in batches for loop filtering but using the same filtering parameters. Multiple first regions divided into the same image can be loop filtered in sequence, using the same or different filtering parameters.

[0088] In one example of this embodiment, the scale factor is a sequence-level or picture-level syntax element. When selecting a scale factor for the first region, the scale factor is selected from candidate scale factors used in the sequence or picture to which the first region belongs. For example, when the scale factor is a sequence-level parameter, the scale factors used by all first regions in multiple pictures included in a sequence are selected from the candidate scale factors determined for the sequence; when the scale factor is a picture-level parameter, the scale factors used by all first regions in a picture are selected from the candidate scale factors determined for the picture.

[0089] Although the above neural network is based on a residual neural network as an example, it is easy to understand that whether the neural network adopts residual connection does not affect the implementation of region-based loop filtering. The region-based loop filtering in this embodiment can also be implemented on a non-residual neural network.

[0090] In an example of this embodiment, this embodiment directly divides the reconstructed image into multiple first regions.

[0091] In another example of this embodiment, the one or more first regions are formed by merging a plurality of second regions into which the reconstructed image is divided, and the second regions include one or more coding tree units. Specifically, the reconstructed image can be divided into one or more first regions by the following steps: S1, dividing the reconstructed image into a plurality of second regions, wherein the second regions with consecutive indexes are adjacent to each other, and each second region includes one or more coding tree units; S2, starting from the second region with the smallest index, and subjecting the index of the merged second region to the constraint that the index should be continuous, determining possible merging methods between the plurality of second regions; S3, determining the merging method with the lowest rate-distortion cost among the possible merging methods, and merging the plurality of second regions according to the determined merging method to obtain the one or more first regions.

[0092] Referring to Figure 6, the image shown is a video frame. Each small, thin square in the figure represents a CTU. A CTU consists of one luma CTB and two chroma CTBs, and the CTB size can be, for example, 128x128. Each thick rectangular box in the image represents a second region. The figure shows 16 second regions indexed from 0 to 15. The sizes of the multiple second regions can be the same or different. In the example shown in Figure 6, 12 second regions each include 6 CTUs, divided into 3 columns. The 4 second regions near the right border have an additional column of CTUs, each including 8 CTUs. This is merely an example. Second region-related parameters, such as the number and position of CTUs included in the second region, can be fixed values ​​configured at the encoder and decoder. The encoder can also encode the relevant parameters into the bitstream, and the decoder can obtain these parameters through decoding. In one example, the number of CTUs included in the second region is a fixed value. This means that images of different resolutions can contain different numbers of second regions, and images with higher resolutions can be divided into more second regions. In another example, the number of the second regions into which the image is divided is a fixed value, which means that the sizes of the second regions into which images with different resolutions are divided are different.

[0093] When numbering the indexes of the second regions, ensure that any two second regions with consecutive indexes are adjacent to each other. In the example shown in Figure 6, starting with the second region with the smallest index, every four second regions with consecutive indexes are located within a larger rectangular region. This reduces the distance between second regions with smaller index differences, making it easier to merge adjacent second regions.

[0094] FIG7 shows the process of merging the 16 second regions in FIG6 . The merging starts from the second region with the smallest index (index 0), and the indexes of the merged second regions are required to be continuous. There is no limit on the number of merged second regions, resulting in a variety of possible combinations. The rate-distortion costs of these merging methods are compared, and the merging method with the smallest rate-distortion cost is used for merging. If the rate-distortion cost of merging all the second regions into one first region is the smallest, the reconstructed image is divided into one first region. The example shown in FIG7 is to merge 16 second regions into 5 first regions, and the 5 first regions include 4, 2, 5, 2 and 3 second regions respectively. When determining the filtering parameters after the merging, the filtering parameters (such as the scale factor α) of the multiple second regions included in the same first region are selected or calculated as a whole.

[0095] The manner in which the reconstructed image is divided into one or more first regions in this embodiment is not limited to the manner shown in Figures 6 and 7. In another example of this embodiment, the reconstructed image is divided into one or more first regions using a recursive partitioning approach. Recursive partitioning herein means that after the image is divided into multiple regions, the previously divided regions are further divided, and so on, until the divided regions are already set non-dividable regions or no longer meet the conditions for further partitioning, as shown in Figure 8. A partitioning decision is made for the reconstructed image or each region larger than a coding tree unit into which the reconstructed image is divided, i.e., determining whether the rate-distortion cost of loop filtering the multiple regions into which the region is divided is less than the rate-distortion cost of directly loop filtering the region. If so, multi-tree partitioning is performed on the region; if not, multi-tree partitioning is not performed on the region, until no more divisible regions larger than the coding tree unit exist, i.e., the coding tree unit is set as non-dividable.

[0096] The example shown in Figure 8 is a recursive quadtree partitioning of an image, but other multi-tree partitioning methods such as binary trees, hexatrees, etc. can also be used. When partitioning, it is first determined whether the rate-distortion cost of loop filtering the image into four regions is less than the rate-distortion cost of directly loop filtering the image. If so, the image is divided into four regions. A similar judgment is made for each divided region, and this recursive process is repeated until no more regions can be divided. Similar to the previous embodiment, the first divided region is at least a basic filtering unit, such as a 128x128 CTU.

[0097] In one embodiment, the neural network used to implement loop filtering is a residual neural network and a scaling factor is used to scale the global residual. The residual neural network can adopt, for example, the CNNLF network model shown in FIG3B and FIG4B but is not limited thereto.

[0098] The operation of scaling the global residual using the scaling factor can be expressed as follows:

[0099] α×P NN +P rec →P org

[0100] Among them, P NN is the global residual of the residual neural network, P rec is the input image of the residual neural network, P org is the reconstructed image after loop filtering.

[0101] The proportional factor α of this embodiment is calculated by the following formula:

[0102]

[0103] The various parameters in the formula are calculated according to linear regression, as follows:

[0104]

[0105]

[0106]

[0107]

[0108] Where i∈Ω represents each pixel i in each image (taking a video frame as an example), and n represents the total number of pixels. NN,i is the global residual of pixel i, P org,i is the value of pixel i in the reconstructed image after loop filtering, P rec,i is the value of pixel i in the input image. The value of pixel i can be calculated separately according to the three color components to obtain the scaling factor used for loop filtering of each component.

[0109] In this embodiment, the scale factor α calculated for each frame can be different. After being integerized, the value of α can be written into the bitstream as a frame-level parameter. The scale factor α can bring additional improvements to encoding performance without increasing decoding complexity, and also makes CNNLF more generalizable. However, the approach of corresponding to a scale factor for each frame is not flexible enough, which limits further improvements in overall encoding performance.

[0110] To this end, an embodiment of the present disclosure provides a loop filtering method, as shown in FIG9 , including:

[0111] Step 210: input the reconstructed image into the residual neural network for loop filtering;

[0112] In this step, when the reconstructed image is input into the residual neural network for loop filtering, image-based, region-based, or block-based loop filtering can be performed. The region-based loop filtering can be implemented using any of the region-based loop filtering embodiments described above.

[0113] Step 220: scaling the global residual of the residual neural network using a scaling factor, wherein the scaling factor is selected from N candidate scaling factors, where N is a variable greater than or equal to 1 or a fixed value greater than 1;

[0114] The number N of candidate scale factors can be used as a parameter at the sequence level or the image level.

[0115] Step 230: Add the scaled global residual to the reconstructed image input into the residual neural network, and output the reconstructed image after loop filtering.

[0116] The loop filtering method of this embodiment can be used on both the encoding and decoding ends and can be implemented based on, but not limited to, the residual neural network shown in Figures 3B and 4B. In this embodiment, the scale factor for scaling the global residual can be selected from N candidate scale factors, where N is a variable greater than or equal to 1 or a fixed value greater than 1. That is, each image, region, or block can have one or more scale factors to choose from. This expands the range of scale factors available, provides greater flexibility in use, and helps improve encoding performance.

[0117] In an example of this embodiment, N is a variable corresponding to the image resolution. The larger N is, the larger the corresponding image resolution is. The number of candidate scaling factors in this example can vary with the change of image resolution. One N value can correspond to one or more image resolutions, but different Ns correspond to different image resolutions. In an image with a larger resolution, such as 3840x2160 (4K), the number of pixels is 83 times that of an image with a smaller resolution, such as 416x240 (240p). Compared with low-resolution images, for example, only one scaling factor is used for one frame, a high-resolution image or a sequence of multiple images uses more candidate scaling factors, which can make the scaling factor better adapt to image changes and improve encoding performance. During the loop filtering process, the encoding end determines the number of candidate scaling factors based on the image resolution of the reconstructed image and the correspondence between N and the image resolution.

[0118] In another example of this embodiment, N is a variable corresponding to a quantization parameter. The larger N is, the smaller the corresponding quantization parameter is. In this example, the number of candidate scale factors is associated with the size of the quantization parameter. One N value can correspond to one or more quantization parameters, but different N values ​​correspond to different optimal parameters. In rate-distortion optimization for loop filtering, in order to achieve a balance between distortion and bit rate, the encoder can use the following formula to measure the cost of the encoding process:

[0119] Cost = D + λ × R

[0120] Where D is the distortion, which is the difference between the reconstructed pixels after loop filtering and the original pixels (based on block, region, or image evaluation); R is the number of bits required to encode the corresponding mode parameters (here, the filtering parameters used for loop filtering) in the current coding mode; λ is a variable related to the quantization parameter. Generally, the λ value is small when the quantization accuracy is high, that is, the quantization parameter is small, and vice versa.

[0121] When there are many candidate scale factors, the number of bits they consume also increases. This has a greater impact on the cost when the λ value is large, but has little impact when the λ value is small. Therefore, to achieve a balanced cost, the number N of candidate scale factors for a sequence or image can be linked to the size of the quantization parameter. When the quantization parameter is large, N can be selected to be small; when the quantization parameter is small, N can be selected to be large.

[0122] In this example, during the loop filtering process, the encoder determines the number of candidate scale factors based on the quantization parameter used for encoding and reconstructing the image and the correspondence between N and the quantization parameter.

[0123] In another example of this embodiment, N is a variable corresponding to image resolution and a quantization parameter, and different Ns correspond to different combinations of image resolutions and quantization parameters. During the loop filtering process, the encoder determines the number of candidate scale factors based on the image resolution of the reconstructed image, the quantization parameter used to encode the reconstructed image, and the correspondence between N and the combination of image resolution and quantization parameter.

[0124] When N is determined to be a variable corresponding to image resolution and / or quantization parameter, a corresponding number of scale factors can be selected as candidate scale factors for sequences or images with different image resolutions and / or quantization parameters. The candidate scale factors can be selected as specific values ​​or can be statistically determined based on a large number of data sets.

[0125] In an example of this embodiment, the N candidate scaling factors are N shift values, where the shift value is a multiplication value corresponding to a right shift operation. For example, the N candidate scaling factors are 1 / 2, 1 / 2, and 2 ,…,1 / 2 N-1 ,1 / 2 N .

[0126] In order to simplify the decoder calculation, this example uses a shift operation to scale the global residual. The corresponding scaling factor is selected as the multiplier value corresponding to the right shift operation (referred to as the shift value in this article), that is, 1 / 2 n , n=1,2,3,…. When there are 4 available shift values, the candidate scale factors are 1 / 2, 1 / 4, 1 / 8, and 1 / 16. The scaling operation can be completed by right shifting the global residual value to be scaled by 1 bit, 2 bits, 3 bits, and 4 bits. In other examples, 1 / 2, 1 / 4, 1 / 8, and 1 / 32 can also be selected as the 4 shift values, that is, 1 / 2 nThe value of n in [ ] can be discontinuous. The shift value selected by the encoder can be directly represented by a binary shift value or by a shorter bit representation. For example, four shift values ​​can be identified by 2 bits, which serve as the shift value identifier, or index. The decoder parses the shift value from the bitstream or determines the shift value based on the parsed identifier and uses the shift value to perform the global residual scaling operation.

[0127] In another example of this embodiment, the N candidate scale factors are the N scale factors with the highest probability of use, obtained based on statistics of scale factors used for loop filtering of a video dataset. Based on dataset statistics, a set number of video datasets can be input into a residual neural network for loop filtering based on images, regions, or blocks. A scale factor to be used for loop filtering is determined for each image, region, or block (in this case, the scale factor can be determined using the aforementioned calculation method, or the one with the lowest rate-distortion cost can be selected from more than N candidate scale factors). The determined scale factors are recorded, and the probability of use of each scale factor is calculated. The N scale factors with the highest probability of use are determined as the N candidate scale factors for the sequence or image. The probability of use of a particular scale factor can be calculated by dividing the number of times the scale factor is used by the total number of times all scale factors are used. This example selects the N most frequently used scale factors as candidates based on the distribution probability of scale factors obtained from statistics of a large number of video datasets, which can achieve good coding performance in most cases.

[0128] In an example of this embodiment, the N candidate scaling factors are constructed as a candidate list of scaling factors (referred to as a candidate residual list for short); there is one candidate list, or there are multiple candidate lists. Different candidate lists mean that the candidate lists contain different numbers of scaling factors, or that the number of scaling factors contained is the same but the value of at least one scaling factor is different. When the N scaling factors with the highest probability of use obtained statistically are used to construct a candidate list, the length of the candidate list is N, and the scaling factors in the candidate list are sorted in descending order according to the probability of use. In this example, when there are multiple candidate lists, different candidate lists may correspond to different image resolutions, or to different quantization parameters, or to different combinations of image resolutions and quantization parameters.

[0129] In an example of this embodiment, the scaling factor is selected from N candidate scaling factors, including: determining a candidate list including the N candidate scaling factors; and calculating a rate-distortion cost of each scaling factor in the candidate list after being used, and selecting a scaling factor with the smallest rate-distortion cost as the selected scaling factor.

[0130] As mentioned above, the number N of candidate scaling factors can be a variable corresponding to the image resolution, the quantization parameter or a combination thereof. Both the encoding end and the decoding end can search and obtain the candidate list based on the image resolution of the reconstructed image and / or the quantization parameter used to encode the reconstructed image. Taking N as an example where the variable corresponds to the image resolution, multiple candidate lists of different lengths are first set for a variety of possible image resolutions. A candidate list can correspond to one or more image resolutions, but different candidate lists correspond to different image resolutions. When the encoding end or the decoding end performs loop filtering on the reconstructed image, the corresponding candidate list is found based on the image resolution of the reconstructed image, and one of the scaling factors contained in the candidate list is selected to scale the global residual.

[0131] For the decoding end, without searching the candidate list through image resolution and / or quantization parameters, the candidate list can be determined by the index of the candidate list obtained by decoding, or the candidate list can be determined according to the number N of scaling factors of the candidates obtained by decoding (different N corresponds to different candidate lists), and a scaling factor is selected from the candidate list to scale the global residual.

[0132] In one example of this embodiment, the candidate list constructed by N candidate scaling factors is a sequence-level or image-level parameter; regardless of whether the scaling factor adopts a shifted value or a value selected based on statistical results, the encoder and decoder can configure and use a consistent candidate list, and in this case, the N candidate scaling factors do not need to be transmitted through the bitstream.

[0133] In one example of this embodiment, the loop filtering is image-based, region-based, or coding tree block-based loop filtering, wherein the region-based loop filtering can be implemented using any of the aforementioned region-based loop filtering methods. When the N candidate scaling factors are sequence-level parameters, when loop filtering based on an image, region, or coding tree block is performed on a reconstructed image, a scaling factor is selected from a candidate list used by the sequence to which the reconstructed image belongs; when the N candidate scaling factors are image-level parameters, when loop filtering based on an image is performed on a reconstructed image, a scaling factor is selected from a candidate list used by the reconstructed image; when the N candidate scaling factors are image-level parameters, when loop filtering based on a region or coding tree block is performed on a reconstructed image, a scaling factor is selected from a candidate list used by the reconstructed image, and the selected scaling factor is used to scale the global residual.

[0134] An embodiment of the present disclosure further provides a video encoding method, as shown in FIG10 , including:

[0135] Step 310, obtaining a reconstructed image based on the predicted image and the residual;

[0136] The encoding process performed by the encoding end on the video image includes obtaining a reconstructed image based on the predicted image and the residual, see FIG1B and related descriptions.

[0137] Step 320 , performing loop filtering on the reconstructed image to obtain a reconstructed image after loop filtering, and encoding filtering parameters used for the loop filtering; wherein the loop filtering is performed according to the loop filtering method described in any embodiment of the present disclosure.

[0138] In an example of this embodiment, the loop filtering is performed according to any of the loop filtering methods for scaling the global residual using a scaling factor, where N is a variable greater than or equal to 1. The encoding of the filtering parameters used for the loop filtering includes: encoding the number N of candidate scaling factors when performing loop filtering on the reconstructed image, where N can be encoded into the sequence header of the sequence to which the reconstructed image belongs or the image header of the reconstructed image. For example, if the scaling factor used when encoding the reconstructed image is selected from 4 candidate scaling factors, a binary code of 4 or 3 can be encoded into the sequence header of the sequence to which the reconstructed image belongs or the image header of the reconstructed image. Encoding 3 into the sequence header or image header can save 1 bit of overhead, and the decoding end can parse the sequence header or image header to obtain 3 and then add 1 to obtain the value of N.

[0139] In one example, after encoding the number N of candidate scale factors for loop filtering of the reconstructed image, the method further includes: encoding the N scale factors as candidates when loop filtering the reconstructed image in the sequence header or image header. This example takes into account that the candidate lists configured simultaneously on both the encoder and decoder are limited. In certain scenarios, when the encoder cannot achieve satisfactory performance using the candidate lists configured on both ends, it can select a candidate list not configured on the decoder. In this case, it is necessary to transmit the N candidate scale factors from this candidate list. Therefore, after encoding N in the sequence header or image header, the encoder continues to encode the N candidate scale factors. The decoder decodes the sequence header or image header and obtains the N scale factors from the same candidate list as the encoder. This approach allows for a larger number of candidate lists to be selected for the reconstructed image, providing greater flexibility and improving encoding performance. Because the most commonly used candidate lists are already configured on both the encoder and decoder, the need to transmit candidate scale factors is rare, resulting in minimal additional overhead.

[0140] In an example of this embodiment, the loop filtering is performed according to the loop filtering method in which a scaling factor is used to scale the global residual and a plurality of candidate lists are provided. The encoding of the filtering parameters used for the loop filtering includes: encoding the index of the scaling factor used when loop filtering the reconstructed image, the region in the reconstructed image, or the coding tree block in the reconstructed image. After the decoder decodes and obtains the index, it searches the corresponding candidate list to determine the scaling factor selected for the reconstructed image, the region in the reconstructed image, or the coding tree block in the reconstructed image. The decoder can then use the scaling factor to scale the global residual and perform loop filtering in the same manner as the encoder. Because the index can be the scaling factor used by the image, the region, or the coding tree block, the index can be encoded as a syntax element at the image level, the region level, or the block level.

[0141] In another example of this embodiment, in addition to encoding the index of the candidate list to which the scaling factor used when loop filtering is performed on the reconstructed image or the area in the reconstructed image or the coding tree block in the reconstructed image belongs, and the index of the scaling factor in the candidate list to which it belongs. In this example, the index of the candidate list is also encoded into the bitstream, and after the decoding end decodes and obtains the index, the candidate list used for the reconstructed image can be determined. In the case where the lengths of different candidate lists (i.e., the number N of candidate scaling factors in the candidate list) are different, N can be encoded into the bitstream as the index of the candidate list, and the index of the candidate list can be encoded into the sequence header or the image header as a sequence-level or image-level parameter. After the decoding end decodes the sequence header or the image header and obtains the index, it can determine the candidate list corresponding to the reconstructed image. In other examples, if the candidate list used by the encoding end can be derived based on the image resolution and / or quantization parameter of the reconstructed image, there is no need to encode the index of the candidate list.

[0142] In one example of this embodiment, the loop filtering is performed according to any of the loop filtering methods described above that uses a scaling factor to scale the global residual and uses a shift value as the scaling factor, wherein the shift value is used as the scaling factor to scale the global residual. Accordingly, encoding the filtering parameters used for the loop filtering includes encoding a shift value or an identifier of the shift value used when performing loop filtering on the reconstructed image, a region within the reconstructed image, or a coding tree block within the reconstructed image. For example, when the shift value is 4, a binary code of 3 can be written into the bitstream to represent the shift value, and the decoder parses the 3 and adds 1 to obtain the shift value.

[0143] In one example of this embodiment, when the loop filtering employs the aforementioned region-based loop filtering method and the first region is formed by merging second regions, encoding the filtering parameters used for the loop filtering includes encoding the number of first regions into which the reconstructed image is divided and the size of each first region. Specifically, the size of a first region can be represented by the number M or M-1 of second regions contained in the first region, where M ≥ 1. Using M-1 can reduce coding overhead.

[0144] In one example of this embodiment, the encoding of the filtering parameters used for the loop filtering further includes at least one of the following: encoding an identifier indicating whether the reconstructed image and / or the sequence to which the reconstructed image belongs enables a scaling factor; and encoding an identifier indicating whether the reconstructed image and / or the sequence to which the reconstructed image belongs enables neural network-based loop filtering. When the encoder has multiple loop filtering methods that can be selected, the encoder does not necessarily use the neural network-based loop filtering method. Therefore, it is necessary to identify whether the neural network-based loop filtering method is enabled. Only when it is determined based on the identifier to use the neural network-based loop filtering, it is necessary to decode the relevant parameters of the neural network-based loop filtering. When using the neural network-based loop filtering method, the scaling factor is not necessarily used. In this case, an identifier indicating whether the scaling factor is enabled is used. When the identifier indicates that the scaling factor is enabled, the decoder needs to decode the parameters related to the scaling factor.

[0145] For the three color components, the flag indicating whether to enable the neural network-based loop filtering and the flag indicating whether to enable the scaling factor may be set separately or in a unified manner.

[0146] An embodiment of the present disclosure provides a video decoding method, as shown in FIG11 , including:

[0147] Step 410: Decode the code stream to obtain filtering parameters used when performing loop filtering on the reconstructed image;

[0148] In this step, loop filtering of the reconstructed image can be performed based on the image, region, or coding tree block. For example, performing region-based loop filtering requires decoding of region-level filtering parameters and possibly image-level and sequence-level syntax elements.

[0149] Step 420: Use the obtained filtering parameters to perform loop filtering on the reconstructed image. The loop filtering is performed according to the loop filtering method described in any embodiment of the present disclosure.

[0150] According to the decoding method of this embodiment, the filtering parameters used by the encoder when performing loop filtering on the reconstructed image can be obtained, and the reconstructed image can be loop filtered in the same manner as the encoder, thereby improving the effect of loop filtering.

[0151] In one example of this embodiment, the loop filtering is performed according to any of the aforementioned loop filtering methods for scaling the global residual using a scaling factor, where N is a variable greater than or equal to 1. Decoding the bitstream to obtain filtering parameters used when performing loop filtering on the reconstructed image includes decoding the bitstream to obtain a number N of candidate scaling factors when performing loop filtering on the reconstructed image.

[0152] In one example of this embodiment, the number N of candidate scale factors that can be used for each frame in the sequence is a sequence-level syntax element variable, taking VVC text as an example. The corresponding sequence parameter set RBSP (raw byte sequence payload) syntax rules are shown in Table 1 below:

[0153] Table 1

[0154]

[0155] In the table, sps_num_scaling_factor_minus1 represents the number of candidate scaling factors, N-1. The decoder can decode the syntax element value and add 1 to obtain the value of N. In the table, u(n) represents the corresponding parameter represented by a fixed-length code of length n.

[0156] Each sequence includes a plurality of consecutive images. In another example, the number N of candidate scale factors that can be used for each frame is a syntax element variable at the picture level, and different frames may have different numbers of scale factor candidates.

[0157] In an example of this embodiment, before decoding the bitstream to obtain filtering parameters used when performing loop filtering on the reconstructed image, the method further includes: decoding an identifier in the bitstream indicating whether the reconstructed image or the sequence to which the reconstructed image belongs enables a scaling factor, and if the identifier indicates that the scaling factor is enabled, further decoding the bitstream to obtain filtering parameters related to the scaling factor.

[0158] For example, the decoder needs to decode a sequence-level control flag first. This flag is used to indicate whether the current sequence allows the use of a scaling factor. The corresponding sequence parameter set RBSP syntax is shown in Table 2 below:

[0159] Table 2

[0160]

[0161] The "sps_enable_scaling_factor" in the table indicates whether the current sequence allows the use of scaling factors. For example, a value of 1 indicates that scaling factors are allowed for the current sequence; a value of 0 indicates that scaling factors are not allowed for the current sequence. If this value indicates that scaling factors are allowed for the current sequence, then the "sps_num_scaling_factor_minus1" and other syntax elements related to scaling factors are decoded.

[0162] In one example of this embodiment, the loop filtering is performed according to any of the aforementioned loop filtering methods for scaling the global residual using a scaling factor, where N is a variable greater than or equal to 1. However, the candidate list used by the encoder to reconstruct the image is not configured at the decoder. In this case, the decoding of the bitstream to obtain filtering parameters used for loop filtering of the reconstructed image includes: decoding a sequence header or a picture header in the bitstream, obtaining the number N of candidate scaling factors for loop filtering of the reconstructed image, and, continuing to decode the N candidate scaling factors in the bitstream based on N, to obtain the N candidate scaling factors for loop filtering of the reconstructed image.

[0163] In this example, in addition to using the sequence-level flag to indicate whether the scaling factor is allowed, the picture-level flag sps_enable_scaling_factor is also used to indicate whether the scaling factor is allowed for the current picture. Based on the value of sps_enable_scaling_factor, the decoder determines whether to decode the number of candidate scaling factors for each frame and parses the candidate scaling factors one by one based on the number. The corresponding picture header structure syntax is shown in Table 3 below:

[0164] Table 3

[0165]

[0166] According to the table above, when decoding the picture header structure, if sps_enable_scaling_factor is 1, the decoding device decodes the value of ph_num_scaling_factor_minus1, adds 1 to it, and obtains the number of candidate scaling factors used for the current picture. It then decodes each scaling factor, ph_scaling_factors[i], where i is the index of the scaling factor. This example can convey a specific candidate list to be used by the encoder when performing loop filtering on the reconstructed picture.

[0167] In the above examples, N is a value directly parsed from the bitstream. However, in other examples, N is a variable corresponding to the quantization parameter and / or image resolution. The decoder can derive the value of N by parsing the quantization parameter and / or image resolution in the bitstream and combining the configured correspondence. In this case, the syntax elements sps_num_scaling_factor_minus1 and ph_num_scaling_factor_minus1 related to the number of scaling factors in Tables 1 and 3 can be omitted.

[0168] In one example of this embodiment, the encoder uses a shift value as a scaling factor, and the decoded bitstream obtains filtering parameters used when performing loop filtering on the reconstructed image, including: decoding syntax elements in the bitstream to obtain a shift value or an identifier of the shift value used when performing loop filtering on the reconstructed image, a region within the reconstructed image, or a coding tree block within the reconstructed image. If the identifier of the shift value is obtained through decoding, the shift value is determined based on the identifier.

[0169] For example, the decoding end may directly decode the number of shifted bits K. For example, when each Ctb selects a shift value when performing loop filtering, the corresponding coding tree unit syntax rules are shown in Table 4 below:

[0170] Table 4

[0171]

[0172]

[0173] Among them, ph_cnnlf_enabled_flag indicates whether CNNLF is enabled for the current image, cnnlf_ctb_flag indicates whether CNNLF is enabled for the current Ctb, the first indexes [0], [1], and [2] after cnnlf_ctb_flag are the indices of the three color components Y, U, and V, respectively, the second index [CtbAddrX] is the horizontal coordinate index of Ctb, and the third index [CtbAddrY] is the vertical coordinate index of Ctb. "ae(v)" in the table is the context model identifier.

[0174] According to the above table, the process of parsing the shift value of each Ctb is as follows:

[0175] Decode ph_cnnlf_enabled_flag. If ph_cnnlf_enabled_flag is 1, it indicates that cnnlf is enabled for the current image. Continue decoding cnnlf_ctb_flag[0][CtbAddrX][CtbAddrY]. ​​If cnnlf_ctb_flag[0][CtbAddrX][CtbAddrY] is 1, it indicates that cnnlf is enabled for the current luminance component Ctb. Continue decoding cnnlf_ctb_scaling_shift[0][CtbAddrX][CtbAddrY] to obtain the shift value of the current luminance component Ctb, which can also be the identifier (or index) of the shift value. The parsing process of the shift values ​​of the current two chrominance components Ctb is similar and will not be repeated here.

[0176] After the decoder obtains the shift value K, the filtering process using the shift value K is as follows.

[0177] P rec ′=P NN >>K+P rec

[0178] Among them, P NN is the global residual of the residual neural network, P rec is the input of the residual neural network, P rec ' is the final output of the residual neural network, and ">>" represents a right shift operation. The filtering process can be performed based on images, regions, or coding tree blocks.

[0179] In one example of this embodiment, referring to Table 4 above, before decoding the bitstream to obtain the filtering parameters used when performing loop filtering on the reconstructed image, the method further includes: decoding an identifier in the bitstream indicating whether neural network-based loop filtering is enabled for the reconstructed image and / or the sequence to which the reconstructed image belongs; and if the identifier indicates enabled, further decoding the bitstream to obtain the filtering parameters used when performing loop filtering on the reconstructed image. In addition to being set as a syntax element at the sequence level or the image level, the identifier indicating whether neural network-based loop filtering is enabled can also be set as a syntax element at the region level or the block level.

[0180] In one example of this embodiment, when the index of a scaling factor used during loop filtering in a candidate list is encoded, decoding the code stream to obtain filtering parameters used when performing loop filtering on the reconstructed image includes: decoding syntax elements in the code stream, obtaining the index of the scaling factor used when performing loop filtering on the reconstructed image, a region in the reconstructed image, or a coding tree block in the reconstructed image, in the candidate list to which it belongs, and determining the scaling factor to be used based on the index and the candidate list (i.e., the candidate list to which the used scaling factor belongs).

[0181] The candidate list to which a scale factor belongs can be determined in a number of ways, for example:

[0182] The first method is to decode the sequence header and image header in the code stream, determine the number N of candidate scale factors when performing loop filtering on the reconstructed image, and determine the candidate list used when performing loop filtering on the reconstructed image based on N; in this method, one N value corresponds to one candidate list.

[0183] The second method is to decode the sequence header and the picture header in the code stream to obtain the index of the candidate list used when loop filtering the reconstructed image, and determine the candidate list used when loop filtering the reconstructed image according to the index;

[0184] The third method involves decoding syntax elements in the bitstream to obtain the image resolution of the reconstructed image and / or the quantization parameter used during encoding. Based on the correspondence between the image resolution and / or quantization parameter and the candidate list, a candidate list is determined for loop filtering of the reconstructed image. This method utilizes the correspondence between the image resolution and / or quantization parameter and the candidate list to derive the candidate list to be used, without requiring the transmission of additional parameters.

[0185] When the third method is used, the decoder parses information such as image resolution and / or quantization parameters from the bitstream, determines an available candidate list, and then obtains the scaling factor from the candidate list based on the index of the scaling factor used by the encoder parsed from the bitstream in the candidate list.

[0186] In one example of this embodiment, when the scaling factor is a value selected based on statistical results, a candidate list can be constructed using the selected scaling factors. The decoder determines the scaling factor selected by the encoder by decoding the index of the scaling factor selected by the encoder. The corresponding coding tree unit syntax is shown in Table 5 below:

[0187] Table 5

[0188]

[0189] In the above table, the meanings of ph_cnnlf_enabled_flag, cnnlf_ctb_flag and their indexes are the same as those in Table 4. cnnlf_ctb_scaling_idx is the index of the scaling factor used by the current color component Ctb in the candidate list of scaling factors.

[0190] In this example, the candidate list ScaleCand consists of several quantized scale factors. The quantized scale factor Scale=ScaleCand[cnnlf_ctb_scaling_idx] is an integer quantized from a floating point number between 0 and 1.

[0191] P rec ′=(P NN ×Scale+1<<(Shift-1))>>Shift)+P rec

[0192] In the above formula, P NN ×Scale plus "1<<(Shift-1)" and then right shifting Shift is the inverse quantization of Scale. If Scale is not quantized to an integer, you can directly use the following formula to calculate:

[0193] P rec ′=P NN ×Scale+P rec

[0194] In an example of this embodiment, when the loop filtering adopts the above-mentioned region-based loop filtering method and the first region is formed by merging the second region, the decoding bitstream obtains the filtering parameters used when performing loop filtering on the reconstructed image, including: decoding the syntax elements in the bitstream, obtaining the number of first regions into which the reconstructed image is divided and the size of each first region.

[0195] When a first region is formed by merging second regions, the encoder and decoder can agree on a rule for dividing the reconstructed image into second regions, so that the decoder can independently deduce the number of second regions and the position of each second region. However, the decoder cannot know the number and position of the merged first regions. Therefore, the encoder encodes the number of first regions and the size of each first region, and the decoder can determine the position of each first region in the image, i.e., the pixels it contains, based on the parsed number of first regions and the size of each first region. Since the first region is merged starting from the second region with the smallest index according to the index continuity constraint, the position of each first region in the image can be determined by obtaining the number of second regions contained in each first region.

[0196] In one example of this embodiment, the reconstructed image is divided into first regions, and the number and size of the first regions are written into the bitstream. The decoding end obtains the number and size of the first regions into which the reconstructed image is divided by decoding the convolutional neural network loop filter data, thereby determining the position of each first region. The syntax rules of CNNLF data (Convolutional neural network loop filter data syntax) are shown in the following table:

[0197] Table 6

[0198]

[0199]

[0200] In Table 6:

[0201] The syntax elements cnnlf_luma_filter_signal_flag, cnnlf_cb_filter_signal_flag, and cnnlf_cr_filter_signal_flag are flags indicating whether the reconstructed images of the Y, Cb, and Cr components are enabled with cnnlf.

[0202] The syntax elements cnnlf_luma_regions_minus1, cnnlf_cb_regions_minus1, and cnnlf_cr_regions_minus1 represent the number of first regions into which the reconstructed image of the Y, Cb, and Cr components is divided, minus 1.

[0203] The syntax elements cnnlf_luma_region_scaling_idx[i], cnnlf_cb_region_scaling_idx[i], and cnnlf_cr_region_scaling_idx[i] respectively represent the index in the candidate list of the scaling factor used for the i-th first region in the reconstructed image of the Y, Cb, and Cr components. When the value of the scaling factor is a shift value, the index can also be represented by the shift value.

[0204] The syntax elements cnnlf_luma_region_distance[i], cnnlf_cb_region_distance[i], and cnnlf_cr_region_distance[i] respectively represent the size of the i-th first region in the reconstructed image of the Y, Cb, and Cr components, and can be represented by the number of second regions contained in the first region minus 1.

[0205] The above CNNLF data can be used as a picture-level syntax element in the code stream.

[0206] An embodiment of the present disclosure further provides a loop filtering device, as shown in FIG12 , comprising a processor 71 and a memory 73 storing a computer program, wherein the processor 71 can implement the loop filtering method as described in any embodiment of the present disclosure when executing the computer program.

[0207] An embodiment of the present disclosure further provides a video decoding device, see FIG12 , comprising a processor and a memory storing a computer program, wherein the processor can implement the video decoding method as described in any embodiment of the present disclosure when executing the computer program.

[0208] An embodiment of the present disclosure further provides a video encoding device, see FIG12 , comprising a processor and a memory storing a computer program, wherein the processor can implement the video encoding method as described in any embodiment of the present disclosure when executing the computer program.

[0209] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes the video encoding device as described in any embodiment of the present disclosure and the video decoding device as described in any embodiment of the present disclosure.

[0210] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program implements the method described in any embodiment of the present disclosure when executed by a processor.

[0211] An embodiment of the present disclosure further provides a code stream, wherein the code stream is generated by the video encoding method as described in any embodiment of the present disclosure.

[0212] The video encoding device and / or video decoding device of the above-described embodiments of the present disclosure may be implemented using any one or any combination of the following circuits: one or more microprocessors, digital signal processors, application-specific integrated circuits, field programmable gate arrays, discrete logic, or hardware. If the present disclosure is partially implemented in software, instructions for the software may be stored in a suitable non-volatile computer-readable storage medium, and one or more processors may be used to execute the instructions in hardware to implement the methods of the embodiments of the present disclosure.

[0213] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, such as according to a communication protocol. In this manner, a computer-readable medium may generally correspond to a non-transitory tangible computer-readable storage medium or a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0214] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may also be referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient (transient) media, but rather refer to non-transient tangible storage media. As used herein, disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, or Blu-ray disc, among others, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0215] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0216] The technical solutions of the embodiments of the present disclosure can be implemented in a wide variety of devices or equipment, including wireless mobile phones, integrated circuits (ICs), or a group of ICs (e.g., chipsets). Various components, modules, or units are described in the embodiments of the present disclosure to emphasize the functional aspects of the devices configured to perform the described techniques, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units can be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

Claims

1. A loop filtering method, comprising: The reconstructed image is input into the neural network for region-based loop filtering; Obtaining a reconstructed image after loop filtering from the output of the neural network; The region-based loop filtering is to divide the reconstructed image into one or more first regions, determine filtering parameters based on the first regions and perform loop filtering, wherein the first regions include one or more coding tree units.

2. The loop filtering method according to claim 1, wherein: The one or more first regions are formed by merging a plurality of second regions into which the reconstructed image is divided, and the second regions include one or more coding tree units.

3. The loop filtering method according to claim 2, wherein: The step of dividing the reconstructed image into one or more first regions comprises: Dividing the reconstructed image into a plurality of second regions, wherein the second regions with consecutive indexes are adjacent to each other, and each of the second regions includes one or more coding tree units; Starting from the second region with the smallest index, determining possible merging methods among the plurality of second regions with the constraint that the indices of the merged second regions should be continuous; Determine a merging manner with the lowest rate-distortion cost among the possible merging manners, and merge the multiple second regions according to the determined merging manner to obtain the one or more first regions.

4. The loop filtering method according to claim 2, wherein: The step of dividing the reconstructed image into one or more first regions comprises: The reconstructed image is divided into one or more first regions by recursive partitioning, wherein a partitioning decision is performed on the reconstructed image or each region larger than the coding tree unit into which the reconstructed image is divided, that is, it is determined whether the rate-distortion cost of loop filtering the multiple regions into which the region is divided is less than the rate-distortion cost of directly performing loop filtering on the region; if so, the region is divided into a multi-branch tree; if not, the region is not divided into a multi-branch tree, until no region larger than the coding tree unit exists that can be divided.

5. The loop filtering method according to claim 1, wherein: The neural network is a residual neural network, the filtering parameters include a scaling factor, and the region-based loop filtering includes: For each first region, the reconstructed image of the first region is input into the residual neural network, the global residual of the residual neural network is scaled using a scaling factor selected for the first region, and the scaled global residual is added to the reconstructed image of the first region input into the residual neural network to obtain the reconstructed image of the first region after loop filtering.

6. A loop filtering method, comprising: The reconstructed image is input into the residual neural network for loop filtering; Scaling the global residual of the residual neural network using a scaling factor, wherein the scaling factor is selected from N candidate scaling factors, where N is a variable greater than or equal to 1 or a fixed value greater than 1; The scaled global residual is added to the reconstructed image input to the residual neural network, and the reconstructed image after loop filtering is output.

7. The loop filtering method according to claim 6, wherein: N is a variable corresponding to image resolution, and the larger N is, the larger the corresponding image resolution is; or N is a variable corresponding to a quantization parameter, and the larger N is, the smaller the corresponding quantization parameter is; or The N is a variable corresponding to the image resolution and the quantization parameter, and different N corresponds to different combinations of image resolution and quantization parameter.

8. The loop filtering method according to claim 6, wherein: The N candidate scale factors are N shift values, and the shift value is a multiplication value corresponding to a right shift operation.

9. The loop filtering method according to claim 6, wherein: The N candidate scale factors are N scale factors with the highest probability of being used, which are obtained based on statistics of scale factors used for loop filtering of the video data set.

10. The loop filtering method according to claim 6, wherein: The N candidate scaling factors are constructed into a candidate list of scaling factors; The candidate list is provided with one; or, There are multiple candidate lists, and different candidate lists correspond to different image resolutions, or different candidate lists correspond to different quantization parameters, or different candidate lists correspond to different combinations of image resolutions and quantization parameters.

11. The loop filtering method according to claim 10, wherein: The candidate list is provided with a plurality of items, and the scaling factor is selected from N candidate scaling factors, including: determining a candidate list of scaling factors including the N candidate scaling factors; Calculating the rate-distortion cost of each scaling factor in the candidate list after it is used, and selecting a scaling factor with the smallest rate-distortion cost as the selected scaling factor; The candidate list is obtained by searching based on the image resolution of the reconstructed image and / or the quantization parameter used to encode the reconstructed image; or, is determined based on the index of the candidate list obtained by decoding; or, is determined based on the number N of candidate scaling factors obtained by decoding.

12. The loop filtering method according to claim 6, wherein: The candidate list constructed by the N candidate scaling factors is a sequence-level or image-level parameter; The loop filtering is image-based, region-based, or coding tree block-based loop filtering, wherein the region-based loop filtering is implemented using the method according to any one of claims 1 to 6.

13. A video decoding method, comprising: Decode the code stream to obtain the filtering parameters used when performing loop filtering on the reconstructed image; The reconstructed image is subjected to loop filtering using the obtained filtering parameters, wherein the loop filtering is performed according to the loop filtering method according to any one of claims 1 to 12.

14. The method of claim 13, wherein: The loop filtering is performed according to the loop filtering method according to any one of claims 6 to 12, wherein N is a variable greater than or equal to 1; The decoding code stream is used to obtain filtering parameters used when loop filtering is performed on the reconstructed image, including: Decoding the code stream to obtain the number N of candidate scale factors for loop filtering the reconstructed image; or Decode the bitstream to obtain the image resolution of the reconstructed image and / or the quantization parameter used in encoding, and determine the number N of candidate scale factors when performing loop filtering on the reconstructed image based on the corresponding relationship between the image resolution and / or the quantization parameter and N.

15. The method of claim 13, wherein: The loop filtering is performed according to any of the loop filtering methods described above that uses a scaling factor to scale the global residual, where N is a variable greater than or equal to 1; The decoding of the bitstream obtains filtering parameters used when performing loop filtering on the reconstructed image, including: decoding a sequence header or a picture header in the bitstream to obtain a number N of candidate scale factors when performing loop filtering on the reconstructed image; and continuously decoding the N candidate scale factors in the bitstream based on the N number to obtain the N candidate scale factors when performing loop filtering on the reconstructed image.

16. The method of claim 13, wherein: The loop filtering is performed according to the loop filtering method according to claim 8; The decoding bitstream is used to obtain filtering parameters used when performing loop filtering on the reconstructed image, including: decoding syntax elements in the bitstream, obtaining a shift value or an identifier of the shift value used when performing loop filtering on the reconstructed image, a region in the reconstructed image, or a coding tree block in the reconstructed image.

17. The method of claim 13, wherein: The loop filtering is performed according to the loop filtering method according to claim 10; The method further comprises decoding a syntax element in the code stream, obtaining an index of a scaling factor used when performing loop filtering on the reconstructed image, a region in the reconstructed image, or a coding tree block in the reconstructed image, in a candidate list, and determining the scaling factor to be used based on the index and the candidate list.

18. The method of claim 17, wherein: The decoding code stream obtains filtering parameters used when loop filtering is performed on the reconstructed image, and further includes: Decoding a sequence header and a picture header in a code stream, determining the number N of candidate scale factors when loop filtering the reconstructed image, and determining a candidate list used when loop filtering the reconstructed image based on N; or Decoding a sequence header and a picture header in a code stream to obtain an index of a candidate list used when loop filtering the reconstructed image, and determining the candidate list used when loop filtering the reconstructed image according to the index; or Decode syntax elements in the bitstream to obtain an image resolution of the reconstructed image and / or a quantization parameter used during encoding, and determine a candidate list to be used when performing loop filtering on the reconstructed image based on a correspondence between the image resolution and / or quantization parameter and the candidate list.

19. The method of claim 13, wherein: The loop filtering is performed according to the loop filtering method according to any one of claims 1 to 5; The decoding bitstream obtains filtering parameters used when loop filtering is performed on the reconstructed image, including: decoding syntax elements in the bitstream, obtaining the number of first regions into which the reconstructed image is divided and the size of each first region.

20. The method of claim 13, wherein: Before obtaining filtering parameters used for loop filtering of the reconstructed image by decoding the code stream, the method further includes at least one of the following: Decoding an identifier in the bitstream indicating whether neural network-based loop filtering is enabled for the reconstructed image and / or the sequence to which the reconstructed image belongs, and if the identifier indicates that loop filtering is enabled, further decoding the bitstream to obtain filtering parameters used when loop filtering is performed on the reconstructed image; The decoded bitstream contains an identifier indicating whether the reconstructed image and / or the sequence to which the reconstructed image belongs enables the scale factor. If the identifier indicates that the scale factor is enabled, the bitstream is decoded again to obtain filtering parameters related to the scale factor.

21. A video encoding method, comprising: Obtain a reconstructed image based on the predicted image and the residual; Perform loop filtering on the reconstructed image to obtain a reconstructed image after loop filtering, and encode filtering parameters used in the loop filtering; wherein the loop filtering is performed according to the loop filtering method according to any one of claims 1 to 12.

22. The method of claim 21, wherein: The loop filtering is performed according to the loop filtering method according to any one of claims 6 to 12, wherein N is a variable greater than or equal to 1; The encoding of the filtering parameters used for the loop filtering includes encoding the number N of candidate scale factors when performing loop filtering on the reconstructed image.

23. The method of claim 22, wherein: After N is encoded into a sequence header of a sequence to which the reconstructed image belongs or an image header of the reconstructed image, and the number N of candidate scaling factors when loop filtering is performed on the reconstructed image is encoded, the method further includes: encoding the N scaling factors as candidates when loop filtering is performed on the reconstructed image in the sequence header or image header.

24. The method of claim 21, wherein: The loop filtering is performed according to the loop filtering method according to claim 10; The encoding of the filtering parameters used for the loop filtering includes: encoding the index of a scaling factor used when performing loop filtering on the reconstructed image, the region in the reconstructed image, or the coding tree block in the reconstructed image, in a corresponding candidate list; or encoding the index of a candidate list to which the scaling factor used when performing loop filtering on the reconstructed image, the region in the reconstructed image, or the coding tree block in the reconstructed image belongs, and the index of the scaling factor in the corresponding candidate list.

25. The method of claim 21, wherein: The loop filtering is performed according to the loop filtering method according to claim 8; The encoding of the filtering parameters used for the loop filtering includes: encoding a shift value or an identifier of the shift value used when performing loop filtering on the reconstructed image, a region in the reconstructed image, or a coding tree block in the reconstructed image.

26. The method of claim 21, wherein: The loop filtering is performed according to the loop filtering method according to any one of claims 2 to 3; The encoding of the filtering parameters used for the loop filtering includes encoding the number of first regions into which the reconstructed image is divided and the size of each first region.

27. The method of claim 26, wherein: The size of each first region is represented by the number M or M-1 of second regions contained in the first region, where M≥1.

28. The method of claim 21, wherein: The encoding of the filtering parameters used in the loop filtering further includes at least one of the following: encoding an identifier indicating whether a scaling factor is enabled for the reconstructed image and / or the sequence to which the reconstructed image belongs; The encoding indicates an identifier indicating whether the reconstructed image and / or the sequence to which the reconstructed image belongs enables loop filtering based on a neural network.

29. A loop filtering device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, the loop filtering method according to any one of claims 1 to 12 can be implemented.

30. A video decoding device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the video decoding method according to any one of claims 13 to 20.

31. A video encoding device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the video encoding method according to any one of claims 21 to 28.

32. A video encoding and decoding system, wherein: It includes the video encoding device as claimed in claim 31 and the video decoding device as claimed in claim 30.

33. A non-transitory computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it implements the loop filtering method as described in any one of claims 1 to 12, or implements the video decoding method as described in any one of claims 13 to 20, or implements the video encoding method as described in any one of claims 21 to 28.

34. A code stream, wherein The code stream is generated by the video encoding method according to any one of claims 21 to 28.

Images