Network-based image filtering for video coding

Abstract

Methods and apparatuses for image filtering in video coding using neural networks are provided.SOLUTION: A method and apparatus includes loading a neural network with a plurality of quantization parameter (QP) map (QpMap) values in a plurality of QpMap channels, obtaining a QP scaling factor by adjusting a plurality of input QP values associated with an input frame, and adjusting the plurality of QpMap values for the neural network according to the QP scaling factor to cause the neural network to learn and filter the input frame.SELECTED DRAWING: Figure 19

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This patent application claims priority to U.S. Provisional Application No. 63 / 182,923, filed May 1, 2021, entitled "Neural Network Based Image Filtering for Video Coding," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to video coding, and more particularly, but not exclusively, to a method and apparatus for video coding using neural network-based model filtering. [Background technology]

[0003] Various video coding techniques can be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include Generic Video Coding (VVC), Joint Exploration Model (JEM), High Efficiency Video Coding (H.265 / HEVC), Advanced Video Coding (H.264 / AVC), Moving Picture Experts Group (MPEG) coding, etc. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit redundancy present in a video image or sequence. An important goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality.

[0004] The first version of the HEVC standard was finalized in October 2013 and provides approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard, H.264 / MPEG AVC. While the HEVC standard offers significant coding improvements over its predecessor, there is evidence that better coding efficiency can be achieved with additional coding tools than HEVC. Based on this, VCEG and MPEG have begun work on developing new coding techniques for future video coding standards. A Joint Video Research Team (JVET) was formed by ITU-T VECG and ISO / IEC MPEG in October 2015 to begin significant research into advanced technologies that could significantly improve coding efficiency. A reference software project, called the Joint Research Model (JEM), was maintained by JVET by integrating several additional coding tools into the HEVC Test Model (HM).

[0005] A Joint Statement for a Video Compression Proposal (CfP) with capabilities beyond HEVC was published by ITU-T and ISO / IEC. 23 CfP responses were received and evaluated at the 10th JVET meeting, demonstrating an approximately 40% increase in compression efficiency over HEVC. Based on these evaluation results, a new project was launched to develop a new generation video coding standard named Versatile Video Coding (VVC). A single reference software code base, called the VVC Test Model (VTM), was established to demonstrate a reference implementation of the VVC standard. Summary of the Invention

[0006] This disclosure provides example techniques related to improving video coding efficiency by using neural network-based model filtering.

[0007] According to a first aspect of the present disclosure, there is provided a method for image filtering in video coding using a neural network, the method including: loading a plurality of quantization parameter (QP) map (QpMap) values ​​in one or more QpMap channels into a neural network, obtaining a QP scaling factor by adjusting a plurality of input QP values ​​associated with an input frame, and adjusting the plurality of QpMap values ​​for the neural network according to the QP scaling factor to train and filter the input frame with the neural network.

[0008] According to a second aspect of the present disclosure, there is provided a method for image filtering in video coding using a neural network, the method including: generating, by a deblocking filter, a deblocking strength map indicating prediction or split block boundaries; and inputting the deblocking strength map into a neural network to train and filter input frames.

[0009] According to a third aspect of the present disclosure, there is provided an apparatus for image filtering in video coding using neural networks, the apparatus including one or more processors and a memory configured to store instructions executable by the one or more processors, the one or more processors configured, upon execution of the instructions, to perform a method according to the first aspect.

[0010] According to a fourth aspect of the present disclosure, there is provided an apparatus for image filtering in video coding using neural networks, the apparatus including one or more processors and a memory configured to store instructions executable by the one or more processors, wherein the one or more processors are further configured, upon execution of the instructions, to perform a method according to the second aspect.

[0011] According to a fifth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform a method according to the first aspect.

[0012] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer-executable instructions which, when executed by the one or more computer processors, cause the one or more computer processors to perform a method according to the second aspect. [Brief explanation of the drawings]

[0013] A more particular description of the embodiments of the present disclosure will be given by reference to specific embodiments that are illustrated in the accompanying drawings, which should be considered to illustrate only some embodiments and, therefore, not to be considered limiting in scope, the embodiments being described and explained with additional specificity and detail through the use of the accompanying drawings. [Figure 1] FIG. 1 is a block diagram illustrating a block-based video encoder according to some implementations of this disclosure. [Figure 2] FIG. 2 is a block diagram illustrating a block-based video decoder according to some implementations of this disclosure. [Figure 3A] FIG. 3A is a schematic diagram illustrating a quadtree splitting mode according to some implementations of the present disclosure. [Figure 3B] FIG. 3B is a schematic diagram illustrating a vertical bisection tree splitting mode according to some implementations of the present disclosure. [Figure 3C] FIG. 3C is a schematic diagram illustrating a horizontal bipartition tree splitting mode according to some implementations of the present disclosure. [Figure 3D] FIG. 3D is a schematic diagram illustrating a vertical 3-tree splitting mode according to some implementations of the present disclosure. [Figure 3E]FIG. 3E is a schematic diagram illustrating a horizontal 3-tree splitting mode according to some implementations of the present disclosure. [Figure 4] FIG. 4 illustrates a simple FC-NN consisting of an input layer, an output layer, and multiple hidden layers, according to some implementations of the present disclosure. [Figure 5A] FIG. 5A illustrates an FC-NN with two hidden layers according to some implementations of the present disclosure. [Figure 5B] FIG. 5B illustrates an example of a CNN in which the dimensions of the second hidden layer are [width (W), height (H), depth (Depth)] according to some implementations of the present disclosure. [Figure 6] FIG. 6 illustrates an example of applying a spatial filter using an input image according to some implementations of the present disclosure. [Figure 7A] FIG. 7A illustrates a residual network (ResNet) including residual blocks as elements of the ResNet, which are added element-wise with their inputs by discriminative connections, according to some implementations of the present disclosure. [Figure 7B] FIG. 7B illustrates an example of a ResNet with overlapping residual modules, according to some implementations of the present disclosure. [Figure 8A] FIG. 8A illustrates an example of a ResNet including multiple residual blocks with globally discriminative connections, according to some implementations of the present disclosure. [Figure 8B] FIG. 8B illustrates another example of a ResNet that stacks multiple residual blocks to further improve video coding efficiency, according to some implementations of this disclosure. [Figure 8C] FIG. 8C illustrates another example of ResNet stack single image super-resolution (SISR) by aggregating the output of residual blocks, according to some implementations in this disclosure. [Figure 9] FIG. 9 illustrates an exemplary neural network-based model for performing image filtering for video coding, according to some implementations of the present disclosure. [Figure 10]FIG. 10 illustrates region-based feature map resolution control according to some implementations of the present disclosure. [Figure 11] FIG. 11 illustrates an exemplary QP-independent neural network model according to some implementations of the present disclosure. [Figure 12A] FIG. 12A illustrates an example of an arrangement of collocated QpMap channels according to some implementations of the present disclosure. [Figure 12B] FIG. 12B illustrates an example of an arrangement of collocated QpMap channels according to some implementations of the present disclosure. [Figure 13] FIG. 13 illustrates chroma upsampling in region 2 of a neural network according to some implementations of the present disclosure. [Figure 14] FIG. 14 illustrates luma downsampling in region 2 of a neural network according to some implementations of the present disclosure. [Figure 15] FIG. 15 illustrates luma downsampling in region 1 of a neural network according to some implementations of the present disclosure. [Figure 16A] FIG. 16A illustrates another example of an arrangement of collocated QpMap channels according to some implementations of the present disclosure. [Figure 16B] FIG. 16B illustrates another example of an arrangement of collocated QpMap channels according to some implementations of the present disclosure. [Figure 17] FIG. 17 illustrates one example of element-wise scaling performed on each residual block according to some implementations of the present disclosure. [Figure 18] FIG. 18 is a block diagram illustrating an apparatus for image filtering in video coding using neural networks, according to some implementations of the present disclosure. [Figure 19]FIG. 19 is a flowchart illustrating a process for image filtering in video coding using neural networks, according to some implementations of the present disclosure. [Figure 20] FIG. 20 is a flowchart illustrating a process for image filtering in video coding using neural networks, according to some implementations of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used. For example, it will be apparent to those skilled in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0015] Throughout this specification, references to "one embodiment," "an embodiment," "an example," "some embodiments," "some examples," or similar language implying that a particular feature, structure, or characteristic is described means that it is included in at least one embodiment or example. A feature, structure, element, or characteristic described in connection with one or more embodiments may be applicable to other embodiments unless expressly specified otherwise.

[0016] Throughout this disclosure, the terms "first," "second," "third," etc. are all used solely as nomenclature to refer to related elements, e.g., apparatus, components, compositions, steps, etc., without implying any spatial or chronological order, unless explicitly specified otherwise. For example, a "first device" and a "second device" may refer to two separately formed devices or two parts, components, or operating states of the same device, and may be arbitrarily named.

[0017] The terms "module," "sub-module," "circuit," "sub-circuit," "circuitry," "sub-circuitry," "unit," or "sub-unit" may include memory (shared, dedicated, or group) that stores code or instructions that can be executed by one or more processors. A module may include one or more circuits, with or without stored code or instructions. A module or circuit may include one or more components that are directly or indirectly connected. These components may or may not be physically attached to each other or located adjacent to each other.

[0018] As used herein, the terms "if" or "when" may be understood to mean "on" or "in response to," depending on the context. These terms, when recited in the claims, do not imply that the associated limitation or feature is conditional or optional. For example, a method may include steps: i) when or if condition X exists, function or action X' is performed; and ii) when or if condition Y exists, function or action Y' is performed. The method may be implemented with both the possibility of performing function or action X' and the possibility of performing function or action Y'. Thus, functions X' and Y' may both be performed at different times and for multiple executions of the method.

[0019] A unit or module may be implemented purely by software, purely by hardware, or by a combination of hardware and software. In the case of a purely software implementation, for example, a unit or module may include functionally related code blocks or software components that are directly or indirectly linked to each other to perform a specific function.

[0020] Like HEVC, VVC is built on a block-based hybrid video coding framework. Figure 1 is a block diagram illustrating a block-based video encoder according to some implementations of the present disclosure. In encoder 100, an input video signal is processed block by block (called a coding unit (CU)). In VTM-1.0, a CU is a maximum of 128x128 pixels. However, unlike HEVC, which divides blocks based only on a quad tree, VVC divides one coding tree unit (CTU) into CUs and adapts them to various local characteristics based on a quad / binary / ternary tree. Furthermore, the concept of multiple division unit types in HEVC is eliminated. That is, the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC; instead, each CU is always used as a basic unit for both prediction and transformation without further division. In a multi-type tree structure, one CTU is first divided by a quad tree structure. Then, each quad tree leaf node can be further divided by a binary and ternary tree structure.

[0021] 3A-3E are schematic diagrams illustrating multi-type tree splitting modes according to some implementations of the present disclosure, each showing five split types, including 4-way split (FIG. 3A), 2-way vertical split (FIG. 3B), 2-way horizontal split (FIG. 3C), 3-way vertical split (FIG. 3D), and 3-way horizontal split (FIG. 3E).

[0022] For each given video block, spatial prediction and / or temporal prediction may be performed. Spatial prediction (or "intra prediction") predicts the current video block using pixels from samples of previously coded neighboring blocks (called reference samples) in the same video picture / slice. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion-compensated prediction") predicts the current video block using pixels reconstructed from previously coded video pictures. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. If multiple reference pictures are supported, a reference picture index is additionally sent, which is used to identify which reference picture in the reference picture store the temporal prediction signal comes from.

[0023] After spatial and / or temporal prediction, an intra / inter mode decision circuit 121 in encoder 100 selects the best prediction mode, for example, based on a rate-distortion optimization method. The block predictor 120 is then subtracted from the current video block, and the resulting prediction residual is decorrelated using transform circuit 102 and quantization circuit 104. The resulting quantized residual coefficients are inverse quantized by inverse quantization circuit 116 and inverse transformed by inverse transform circuit 118 to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. Furthermore, further in-loop filtering 115, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive in-loop filter (ALF), may be applied to the reconstructed CU before it is placed into a reference picture store in picture buffer 117 and used to code future video blocks. To form the output video bitstream 114, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit 106 for further compression and packing to form the bitstream.

[0024] For example, deblocking filters are available in AVC, HEVC, as well as the current version of VVC. In HEVC, an additional in-loop filter called SAO is defined to further improve coding efficiency. In the current VVC standard, yet another in-loop filter called ALF is actively investigated and is likely to be included in the final standard.

[0025] These in-loop filter operations are optional. Performing these operations helps improve coding efficiency and visual quality. They may also be turned off as a decision is made by the encoder 100 to save computational complexity.

[0026] Note that intra prediction is typically based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100.

[0027] Figure 2 is a block diagram illustrating a block-based video decoder 200 that can be used with many video coding standards. This decoder 200 is similar to the reconstruction-related section present in the encoder 100 of Figure 1. In the decoder 200, an input video bitstream 201 is first decoded through entropy decoding 202 to derive quantized coefficient levels and prediction-related information. The quantized coefficient levels are then processed through inverse quantization 204 and an inverse transform 206 to obtain a reconstructed prediction residual. A block prediction mechanism implemented in intra / inter mode selection 212 is configured to perform either intra prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by summing the reconstructed prediction residual from the inverse transform 206 and the prediction output generated by the block prediction mechanism using an adder 214.

[0028] The reconstructed blocks may further pass through an in-loop filter 209 before being stored in a picture buffer 213, which serves as a reference picture store. The reconstructed video in the picture buffer 213 may be transmitted to drive a display device, as well as used to predict future video blocks. In situations where the in-loop filter 209 is turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video output 222.

[0029] The present disclosure seeks to improve the image filtering design of the above-mentioned video coding standards or techniques. The filtering method proposed in this disclosure is neural network-based, which can be applied, for example, as part of in-loop filtering between a deblocking filter and a sample adaptive offset (SAO), or as part of post-loop filtering to improve current video coding techniques, or as part of post-processing filtering after current video coding techniques.

[0030] Neural network techniques, such as fully connected neural networks (FC-NNs), convolutional neural networks (CNNs), and long short-term memory networks (LSTMs), have already achieved remarkable success in many research areas, including computer vision and video understanding.

[0031] [Fully connected neural network (FC-NN)] 4 illustrates a simple FC-NN consisting of an input layer, an output layer, and multiple hidden layers, according to some implementations of the present disclosure. k (x (k-1) ,W k ,B k ) is generated by the following formula:

number

number

number

number

[0032] According to the universal approximation hypothesis and equation (4), given any continuous function g(x) and some ε > 0, there exists a neural network f(x) with a nonlinearity, e.g., a reasonable choice of ReLU, such that ∀x, |g(x)-f(x)| < ε. Therefore, many empirical studies have applied neural networks as approximators, mimicking models with hidden variables, to extract explainable features under surfaces. For example, when applied to image recognition, FC-NNs can help researchers build systems that understand not just single pixels but increasingly deeper and more complex substructures, such as edges, textures, geometric shapes, and objects.

[0033] [Convolutional Neural Network (CNN)] FIG. 5A illustrates an FC-NN with two hidden layers according to some implementations of the present disclosure. CNN, a common neural network architecture for image or video applications, is very similar to the FC-NN shown in FIG. 5A, but it includes weights and bias metrics. CNNs can be considered a 3D version of neural networks. FIG. 5B illustrates an example of a CNN in which the dimensions of the second hidden layer are (width (W), height (H), and depth)) according to some implementations of the present disclosure. In FIG. 5B, neurons are arranged in a three-dimensional structure (width, height, and depth) to form a CNN, and the second hidden layer is visualized. In this example, the input layer holds the input image or video frame, so its width and height are the same as the input data. For image or video applications, each neuron in the CNN is a spatial filter element with an extended depth aligned with its input; for example, if there are three color components in the input image, the depth is 3.

[0034] 6 illustrates an example of applying a spatial filter using an input image according to some implementations of the present disclosure. As shown in FIG. 6, the dimensions of the basic elements in CNN are [Filter width , Filter height , Input depth , Output depth ] and set to [5,5,3,4]. Each spatial filter performs a 2D spatial convolution on the input image with 5x5x3 weights. The input image may be a 64x64x3 image. Four convolution results are then output. Therefore, if you pad the borders with an additional 2 pixels, the dimensions of the filtered result are [64+4,64+4,4].

[0035] [Residual Network (ResNet)] In image classification, accuracy saturates and rapidly degrades as the depth of a neural network increases. More specifically, adding more layers to a deep neural network results in higher training errors because gradients gradually vanish along the deep network and eventually approach zero gradient. ResNet, which is composed of residual blocks, solves the degradation problem by introducing discriminative connections.

[0036] 7A illustrates a ResNet including a residual block as an element of the ResNet, which is added element-wise with its input by a discriminative connection unit, according to some implementations of the present disclosure. As shown in FIG. 7A, the basic module of the ResNet consists of a residual block and a discriminative connection unit. According to the universal approximation hypothesis, given an input, a weighted layer with an activation function in the residual block approximates the hidden function F(x) rather than the output H(x)=F(x)+x.

[0037] By stacking nonlinear multi-layer neural networks, the residual block searches for features that represent local characteristics of the input image. Without introducing additional parameters and computational complexity, the discriminant connection proves that the deep learning network can be trained by skipping one or more nonlinear weighted layers, as shown in Figure 7A. When skipping weighted layers, the differential output of the residual layer can be written as follows:

number

[0038] Therefore, the differential term

number

[0039] [ResNet variations] 8A-8B, several variations of ResNet have been proposed to improve the recovered image quality of single image super-resolution (SISR) and increase the accuracy of image classification. FIG. 8A shows an example of a ResNet including multiple residual blocks with global discriminant connections according to some implementations of the present disclosure. In FIG. 8A, variations of ResNet are proposed to improve the visual quality of upsampled images. Specifically, the global discriminant connections are applied from the input of the first residual block to the output of the last residual block to facilitate the convergence of the learning procedure.

[0040] Figure 8B shows another example of a ResNet that stacks multiple residual blocks to further improve video coding efficiency, according to some implementations of this disclosure. Each residual block directly propagates its input to the next unit through a concatenation operation. In other words, each intermediate block can receive multi-layer information from its preceding unit, since multi-level information can flow through the same connection. The parameters of each residual block in Figure 8B increase linearly with the number of layers through the concatenation operation.

[0041] 8A-8B, the residual features must pass through one or several modules before the residual information can be propagated to subsequent modules. Due to discriminative connections, these residual features can quickly merge with discriminative features in a particular layer and stop propagating to subsequent modules. Therefore, the residual features in the previous two variants are locally limited, resulting in performance degradation.

[0042] FIG. 8C illustrates another example of ResNet stacked single-image super-resolution (SISR) by aggregating the outputs of residual blocks according to some implementations of the present disclosure. In FIG. 8C, the output of the last residual block is concatenated with the outputs of all three previous modules. Before applying element-wise summation with the input of the first residual block, the concatenated hierarchical features are fused by a convolution operation. Unlike the first two variations, the aggregated ResNet allows non-local features to be applied to the last residual block, thereby allowing hierarchical information to propagate to subsequent blocks, achieving feature representation in a more discriminatory manner.

[0043] In this disclosure, methods and apparatuses related to neural network-based image filtering are proposed to further improve the coding efficiency of current hybrid video coding. The proposed methods and apparatuses can be applied as part of in-loop filtering, for example, between a deblocking filter and a sample adaptive offset (SAO) as shown in Figure 2, or as post-loop filtering to improve current video coding techniques, or as post-processing filtering after current video coding techniques.

[0044] FIG. 9 illustrates an exemplary neural network-based model for performing image filtering for video coding according to some implementations of the present disclosure. YUV components may be provided to the neural network model in parallel. This parallel input of YUV components may be beneficial not only to reduce processing delay but also for the neural network model to learn correlations between juxtaposed YUV information, such as cross-component filtering and / or luma-guided chroma filtering. The on / off control of this neural network model-based filter may be performed at the coding tree unit (CTU) level for a reasonable trade-off between control granularity and signaling overhead. The on / off control of the neural network model-based filter for the YUV components may be performed at the same or different granularity. For example, the on / off control of this neural network model-based filter for the Y component may be performed at the CTU level, while the on / off control for the U and V components may be performed at the frame level, for example, to save CTU-level flag signaling overhead.

[0045] [Feature map resolution alignment] As shown in Figure 9, when CTU-level YUV information is provided to the neural network model filter, the resolutions of the YUV CTU patches may or may not be the same. For example, if the encoded video content is YUV420, the resolutions of the three juxtaposed YUV patches may not be the same. In this case, resolution alignment is required. For ease of illustration, all methods and devices proposed in this disclosure assume that the video content is YUV420. The proposed methods can be easily extended to different content formats, such as YUV422 and YUV444.

[0046] In some embodiments, resolution alignment may be performed before the YUV patches enter the neural network.

[0047] In some embodiments, one 128x128 Y patch may be downsampled to one 64x64 patch or four 64x64 patches. When four 64x64 patches are generated, all information in the original 128x128 patch may be preserved and distributed among the four patches. The method used to distribute the information from the original 128x128 patch may be partition-based, e.g., one 64x64 patch may be from the top left of the original 128x128 patch, and another 64x64 patch may be from the top right of the original 128x128 patch. Alternatively, the method used to distribute the information from the original 128x128 patch may be interleaved, e.g., every four adjacent samples of the original 128x128 patch are evenly distributed among the four 64x64 patches.

[0048] In some embodiments, one 64x64 U or V patch may be upsampled to one 128x128 patch.

[0049] In some embodiments, resolution alignment may be performed after the YUV patches enter the neural network. In one embodiment, the Y input resolution may be reduced to match the UV input. One way to achieve this is to use a convolutional layer with a stride size twice as large as the UV input. In this embodiment, a resolution increase layer is needed at the end of the neural network to scale up the Y content so that the output of the model has the same resolution as the input. One way to achieve this is to use a pixel shuffle layer to scale up the Y resolution. In another embodiment, the UV input resolution may be increased to match the Y input. One way to achieve this is to use a pixel shuffle layer to scale up the UV at the start of the neural network and then scale it down at the end of the neural network.

[0050] [Feature map resolution control] The feature map resolution proportionally affects the neural network processing overhead, but may not proportionally affect the performance of the neural network. To control the computational complexity of model filtering, various solutions may be available, such as the number of residual blocks and the number of input / output channels of the convolutional layer in each residual block. Controlling the resolution of the feature map in the convolutional layer is another effective option for controlling the computational complexity.

[0051] FIG. 10 illustrates region-based feature map resolution control according to some implementations of the present disclosure. As shown in FIG. 10, three regions can be used to adjust the feature map resolution for computational complexity control (which can be referred to as region-based feature map resolution control). In region 1, the resolution of the input YUV patch is determined and a corresponding scale-up / down operation is performed. For example, an example of an up / downsampling method introduced in "feature map resolution alignment" is shown in FIG. 15.

[0052] FIG. 15 illustrates luma downsampling in region 1 of a neural network according to some implementations of the present disclosure. As shown in FIG. 15, an original Y patch is downsampled to four downsampled Y patches in region 1 before entering the neural network. For example, one 128×128 Y patch may be downsampled to four 64×64 Y patches. An inverse operation, e.g., upsampling, is performed in region 1 after the neural network finishes processing. As shown in FIG. 15, the four downsampled Y patches output by the neural network are upsampled to one original Y patch. For example, four 64×64 Y patches may be upsampled to one 128×128 Y patch.

[0053] In Region 2, the resolution of the input YUV patches is determined, and the corresponding scaling up / down operation is performed before YUV concatenation. Because this region is located at the beginning of the neural network, if a scaling down operation is performed, significant input information may be lost, which may impair the overall performance after model training. Two examples are shown in Figures 13 and 14, respectively.

[0054] 13 illustrates chroma upsampling in region 2 of a neural network according to some implementations of the present disclosure. As shown in FIG. 13, after a UV patch enters the neural network, it is scaled up by the corresponding convolution block or layer in region 2 of the neural network. An inverse operation, e.g., scale down, is performed on the corresponding UV patch output by the last residual block in region 3, as shown in FIG.

[0055] 14 illustrates luma downsampling in region 2 of a neural network according to some implementations of the present disclosure. As shown in FIG. 14, after a Y patch enters the neural network, it is scaled down by the corresponding convolutional block or layer in region 2 of the neural network. An inverse operation, e.g., scale up, is performed on the corresponding Y patch output by the last residual block in region 3, as shown in FIG.

[0056] In Region 3, the resolution of the input YUV patch may be scaled up / down in one of the earlier residual blocks, and the inverse operation, e.g., scale down / up, may be performed in a later residual block. Because this region is located after YUV concatenation, when the scale down operation is performed, the input information may be significantly less lossy than in Region 2 because most of the input information has already been captured or learned in previous convolutional layers with sufficient depth for information learning. For example, after Region 2, three-channel YUV content with UVs scaled up to 128x128 is generated. The Y input information may already have been learned / extracted and distributed / replicated in previous convolutional layers before concatenation. Alternatively, the scale down operation may be performed after the first residual block, since the first residual block may have enough channels to learn / extract Y input features.

[0057] [QP-independent neural network model] To facilitate easier deployment of the proposed neural network model filtering, it is desirable to remove input quantization parameter (QP) dependency from the neural network model. Thus, a single neural network model can be used for image filtering regardless of the input QP used for video coding.

[0058] FIG. 11 illustrates a typical QP-independent neural network model according to some implementations of the present disclosure. In a typical video coding system, a QP value is used to calculate a quantization step size for prediction residual quantization / dequantization. Therefore, different QP values ​​represent different levels of video quality. To handle different video frames with different input QPs and qualities, a QpMap is provided to the neural network. The QpMap adds another dimension of information for the neural network to learn and adaptively filter the provided YUV input, which may include different levels of video quality (e.g., input Qp). In some common video coding standards, such as HEVC, VVC, or AVS, when an input Qp value is converted to a Qp step size for prediction residual quantization, a predetermined relationship (e.g., Q step =2 (QP-4) / 6 ) is commonly used. For easier explanation, we introduce the proposed idea as follows using a QpMap that contains or includes either the input Qp values ​​or the Qp step size values.

[0059] [Dynamic range control of QpMap values] 11 illustrates an exemplary QP-independent neural network-based model for performing image filtering for video coding according to some implementations of the present disclosure. A QpMap is concatenated with YUV input channels. Each QpMap channel may contain the same values. Each QpMap channel may have the same resolution as the associated input channel. That is, the QpMap channel for the Y input has the same resolution as the Y input channel, and each value of the QpMap indicates that all samples in the Y input channel have the same Qp value.

[0060] The input QP values ​​for each video frame / image may be used directly to generate a QpMap value. Alternatively, the input QP values ​​for each video frame / image may be first used to generate a QpMap value, for example, by adjusting a QP step size, e.g., Q step =2(QP-4) / 6 can be converted into

[0061] When QpMap values ​​are generated from an input QP or Qp step size, it is desirable that the QpMap value dynamic range be reasonable in the following three senses.

[0062] First, the range must be large enough so that different QpMap values ​​can be easily used to represent / distinguish different input Qp or Qp step sizes. In other words, given two input Qp values, the corresponding QpMap values ​​must not be close to each other.

[0063] Second, the range should be well balanced so that different QpMap values ​​can be evenly distributed at different positions in the range.

[0064] Third, the range must match the dynamic range of the associated YUV sample values. For example, P max If the YUV sample values ​​are normalized to [0,1] by dividing max =2 bitdepth -1, but the QpMap values ​​are assumed to be normalized to a similar range.

[0065] Therefore, it is proposed that when the QpMap is mapped to the dynamic range, the maximum or minimum input Qp or Qp step size should not be used as a division factor, otherwise the division will push the generated QpMap values ​​towards one side of the dynamic range, which is equivalent to reducing the dynamic range of the QpMap values.

[0066] For example, when using the maximum Qp step size of 912 (corresponding to a maximum input Qp of 63), the theoretical dynamic range is (0, 1), but when the input Qp step size is typically less than 45 (corresponding to an input Qp of 37), the effective dynamic range is only (0, 0.05), which means that in most cases the QpMap value will be close to 0.

[0067] Instead, it is proposed to perform normalization using a mean / median input Qp or Qp step size, so that the generated QpMap values ​​can be distributed on either side of the dynamic range (e.g., [0.5, 1.5]). One exemplary selected mean / median input Qp value is Qp32, and then normalization is performed using the Qp to Qp step (Qp step size) equation (e.g., Q step =2 (QP-4) / 6 ), which is approximately 25.5. Therefore, any input Qp value is first converted to the corresponding Qp step size, followed by division by the selected Qp step size of 25.5.

[0068] The selected input Qp or Qp step size for normalized division can be flexibly determined based on the actual input Qp range. For example, given an actual Qp range [22, 42], Qp 37 or its corresponding Qp step size can be selected as the division factor so that the normalized values ​​of lower Qps in the actual Qp range are not too close to zero, while the normalized values ​​of higher Qps are not too above 1.0. Alternatively, the maximum value of the actual input QP range (e.g., 42 in the actual Qp range [22, 42]) or its corresponding Qp step size can be selected as the division factor if the maximum value is not too larger (e.g., within twice the size) than the minimum value.

[0069] [Adjustment based on predicted QpMap values] As described above, the QpMap value may be generated directly by the input Qp value, or may be generated by the Qp step size value according to the mapping relationship between the input Qp value and the Qp step size. For ease of illustration, the following description assumes that the QpMap value is generated directly by the input Qp value. When the QpMap value is generated by the Qp step size value, the proposed idea / method can be similarly extended.

[0070] For inter-predicted video frames / pictures, most blocks / CTUs in the frame / picture may have small or no residual, e.g., may be inter-predicted in skip mode, in which case the effective input Qp value should be determined by the corresponding reference frame / picture.

[0071] In some embodiments, the input Qp value of the corresponding reference frame / image can be stored and retrieved when the current image is reconstructed during the motion compensation process. The input QP value of each current frame is known. However, when this frame is not current and this frame is a reference frame for another frame, the input Qp value of this frame becomes unknown. Therefore, the Qp value must be stored to retrieve it in the future.

[0072] In some embodiments, the input Qp value of the corresponding reference frame / image may be derived by subtracting a specific value from the QP value of the current frame being inter-coded, and the specific value may be obtained by checking the temporal layer index of the current frame being inter-coded.

[0073] In some other embodiments, if the reference frame / picture is a chain of reference pictures (reference frame / picture of reference frame / picture), this information may be inherited or conveyed from signaling.

[0074] In a simple solution, for an inter-predicted video frame / picture, the effective Qp step size can be derived from the current frame's Qp step size by a constant scaling factor such as 0.5, which corresponds to an input Qp differential with value 6. This scaling operation is an approximation to the reference frame / picture input Qp or Qp step size.

[0075] Qp step size and input Qp (e.g. Q step =2 (QP-4) / 6), the scaling operation of the Qp step size is equivalent to a subtraction / addition operation of the input Qp values. That is, the scaling operation of the Qp step size can be implemented by applying a subtraction / addition operation of the input Qp values.

[0076] Depending on the trade-off between signaling overhead and prediction-based adjustment accuracy, scaling of the Qp step size or subtraction / addition of input Qp values ​​can be expressed with different precision and / or different granularity. For ease of illustration, the ideas / methods proposed below assume that scaling of the QpStep size is used for prediction-based adjustment of QpMap values. When subtraction / addition of input Qp values ​​is used for prediction-based adjustment of QpMap values, the proposed ideas / methods can be directly extended.

[0077] In one or more embodiments, scaling of the Qp step size may also be applied to intra-predicted frames / pictures to compensate for the inaccuracy of the constant scaling factor used for subsequent inter-predicted frames.

[0078] In another embodiment, the Qp scaling factor can be flexibly derived using different methods:

[0079] In the first method, a Qp scaling factor may be selected by the encoder from a set of values. The scaling factor set may be sequence-based or picture / slice-based, meaning that the set may be coded in the picture header or the sequence parameter set. The index of the selected Qp scaling factor in the scaling factor set may be selected based on an encoder-side rate-distortion optimization algorithm with different granularities, e.g., picture-level index selection, CTU-level index selection, block-level selection (e.g., a picture may be divided into different blocks based on quadtree partitioning), for the purpose of a good trade-off between image quality and signaling overhead.

[0080] In the second method, the Qp scaling factor can be converted into a Qp offset / adjustment, which can be applied to the input Qp or Qp step size before calculating the QpMap value.

[0081] In one embodiment, the adjusted input Qp value may be expressed as:

number

[0082] For example, if signaled offset_index is 1, Qp_offset_stepsize can be 4, lower_bound can be a constant value like 2, and the decoder can adjust the current Qp value of 32 to 28, where 28 = 32 - 4 * (2 - 1).

[0083] In the third method, for the current CTU / picture / block, the Qp scaling factor can be inherited from a neighboring CTU / picture / block (e.g., the left or upper CTU / block in the spatial domain, or the reference block / CTU in the temporal domain) to save signaling overhead.

[0084] In a fourth method, instead of signaling or inheritance, the decoder side can calculate the Qp scaling factor according to the Qp difference between the current CTU / block and the reference CTU / block. If there are multiple reference CTUs / blocks corresponding to the current CTU / block, the average value of the Qp scaling factor can be calculated. If the reference CTU / block is included in a reference chain, the reference depth can be constrained, and the Qp scaling factor can be calculated according to at most the parent reference CTU / block of the constrained reference depth.

[0085] In a fifth method, the Qp scaling coefficients of different components can be jointly signaled / selected / calculated for lower complexity. Alternatively, the Qp scaling coefficients of different components can be signaled / selected / calculated separately. Alternatively, the Qp scaling coefficients of luma and chroma can be signaled / selected / calculated separately.

[0086] In a sixth method, any combination of the above methods can be used as a hybrid method.

[0087] [Scaling sample values ​​based on QpMap values] A QP-independent neural network model may not explicitly include a QpMap channel in the network. For example, as shown in Figure 11, a QpMap generated for each YUV channel may be concatenated with the YUV channel. Alternatively, the QpMap values ​​provided to the network may be used to directly scale the sample values ​​in each YUV channel. In this way, although the QpMap channel is not concatenated with the YUV channel, this means that the QpMap is implicitly used in the network.

[0088] When the QpMap channel is injected into the network and concatenated with the YUV channel, sample value scaling, i.e., scaling of sample values ​​in the YUV channel, can be performed directly by element-wise multiplication, as shown in Figure 11. For example, each element of the QpMap channel for the Y component is multiplied with the corresponding element of the Y channel, and each element of the QpMap channel for the U or V component is multiplied with the corresponding element of the U or V channel. Note that the resolution of the QpMap channel may already be aligned with the resolution of the corresponding component channel.

[0089] In another example, element-wise scaling may also be performed in each residual block. Figure 17 illustrates an example of element-wise scaling performed in each residual block according to some implementations of the present disclosure. In Figure 17, the QpMap channel is first concatenated with the YUV channel after the YUV resolutions are aligned. Then, the QpMap channel is used not only as the input feature map for the first residual block, but also as the sample value scaling coefficients in each residual block.

[0090] When sample scaling is performed on each residual block, the scaling operation may be performed at the beginning of the residual block (e.g., the input of the residual block as in FIG. 7A) or at the end of the residual block (e.g., the output of the residual block as in FIG. 7A).

[0091] When sample scaling is performed on each residual block, the QpMap information is equivalent to being reinforced multiple times in the network.

[0092] It should be noted that the above two sample scaling mechanisms may be used exclusively or may be combined. In other words, the sample scaling applied directly to the YUV samples before concatenation, such as in Figure 11, and the sample scaling applied to each residual block, such as in Figure 17, may both be used in the same neural network, or may be used separately in different neural networks.

[0093] In some instances of implicit use of QpMap, the QpMap data may not be fed to the network, and the scaling of the sample values ​​in each YUV channel is performed before the neural network.

[0094] [Interaction between neural network-based model filtering and other in-loop filters] When the QpMap channel is provided to a neural network to filter video content with varying qualities, the QpMap channel may contain Qp information from one or more components.

[0095] Figure 12A illustrates an example of an arrangement of a co-located QpMap channel according to some implementations of the present disclosure. Figure 12B illustrates another example of an arrangement of a co-located QpMap channel and a YUV channel according to some implementations of the present disclosure. In Figures 12A-12B, block Map-Y indicates the QpMap channel for the Y channel, block Map-U indicates the QpMap channel for the U channel, and block Map-V indicates the QpMap channel for the V channel. Blocks Y, U, and V indicate the Y, U, and V channels, respectively.

[0096] Given YUV420 content, the UV components may first be upsampled, and then the YUV may be co-located and interleaved with the corresponding QpMap channels as shown in Figure 12A, or the Y channels may first be downsampled to four smaller Y channels, and then the YUV may be co-located and interleaved with the QpMap channels as shown in Figure 12B. In some embodiments, either the upsampling or downsampling is performed within the network, e.g., within regions 2 and 3 of Figure 10, or outside the network, e.g., within region 1 of Figure 10.

[0097] Figure 16A illustrates another example of an arrangement of co-located QpMap channels according to some implementations of the present disclosure. Figure 16B illustrates another example of an arrangement of co-located QpMap channels according to some implementations of the present disclosure. In Figures 16A-16B, block Map-Y indicates the QpMap channel for the Y channel, block Map-U indicates the QpMap channel for the U channel, and block Map-V indicates the QpMap channel for the V channel. Blocks Y, U, and V indicate the Y, U, and V channels, respectively. The QpMap channels may be first concatenated internally and then concatenated with the YUV channels as shown in Figures 16A-16B.

[0098] In cases where only one or more QpMap channels of a component are provided to the neural network, the one or more QpMap channels may be placed on either side of the YUV channels, indicating that the YUV channels are concatenated prior to the addition of the QpMap channel so that the YUV channels are adjacently juxtaposed.

[0099] In another embodiment, in addition to the QpMap channels from different components, additional QpMap channels for different types of training data may be needed. For example, if the training data is cropped from I frames, B frames, or P frames, a QpMap containing frame type information may be generated and concatenated. I frames are intra-coded frames, B frames are bidirectionally predicted frames, and P frames are predicted frames.

[0100] [Filtering offset or scaling of the output of a neural network model-based filter] For generalization purposes, the unified neural network model-based filter can be used for different video content with different levels of quality, motion, and lighting environments. The output of the neural network model-based filter can be slightly adjusted by forming an offset or scaling at the encoder side for better coding efficiency.

[0101] The filtering offset or scaling value may be adaptively selected by the encoder from a set of values. The offset set or scaling set may be sequence-based or picture / slice-based, meaning that the set may be coded in a picture / slice header or a sequence parameter set. The index of the selected offset or scaling value in the set may be selected based on an encoder-side rate-distortion optimization algorithm with different granularities, e.g., picture-level index selection, CTU-level index selection, block-level selection, e.g., a picture may be divided into different blocks based on a quadtree partitioning for the purpose of a good trade-off between image quality and signaling overhead.

[0102] The selection of adaptive filtering offset or scaling value can be based on a specific classification algorithm, such as the smoothness of the content or a histogram of orientation gradients, etc. The adaptive filtering offset or scaling value for each category is calculated and selected in the encoder and explicitly signaled to the decoder to effectively reduce sample distortion, while the classification of each sample is performed in both the encoder and decoder to significantly save side information.

[0103] The selection of adaptive filtering offsets or scaling values ​​may be performed for different components jointly or separately, for example YUV have different adaptive filtering offsets or scaling values.

[0104] [Generation of training data and training process] When a neural network-based filter model is trained, the training data preparation and training process can be performed in different ways.

[0105] In some examples, the model may be trained based on a dataset with only still images, which may be encoded with all I-frames from a video coding tool in which a neural network-based filter is used.

[0106] In some embodiments, a model may be trained based on a two-pass process. In the first pass, a dataset may be encoded with all I frames, and model A may be trained based on all I frames. In the second pass, the same dataset or a new dataset may be encoded with a combination of I, B, and P frames with a different ratio (ratio of the number of I, B, and P frames included). In some embodiments, the generated I / B / P frames are encoded by applying model A trained in the first pass. A new model B may be trained based on the newly generated I / B / P frames.

[0107] When model B is trained, model A can be loaded as a pre-trained model such that model B is a refined model starting from model A. In another embodiment, another model different from model A can be loaded as a pre-trained point.

[0108] Alternatively, model B can be trained from scratch.

[0109] In some embodiments, a model may be trained multi-pass, with three or more passes. In a first pass, model A may be trained based on I frames. In a second pass, when model A is applied to the encoder, model B may be trained or refined based on model A based on a combination of I / B / P frames. Note that the selected combination of B / P frames in this second training pass may only be from a lower temporal layer. In a third or further pass, model C may be trained or refined based on model B based on a higher temporal layer of B / P frames. When a higher temporal layer of B / P frames is generated and selected, model B or / and model A may be applied at the encoder side.

[0110] Before network training, training data needs to be generated. In this multi-pass method, training data is generated through three passes, including: the first pass generates only I-frames, which are used to train model A; once model A is prepared, the encoder can load model A or not, and generate low-temporal layer B / P frames, called the second pass; these generated low-temporal layer B / P frames are used to train model B through new training or refine based on model A.

[0111] Furthermore, once model B is ready, the encoder may or may not load models A and B, and may generate high-temporal layer B / P frames, called the third pass, which are used to train model C by new training or to refine it based on models A and / or B.

[0112] [Interaction between neural network-based model filtering and other in-loop filters] When neural network-based model filtering is signaled to be on at the CTU level or frame level, deblocking filtering can be skipped to avoid unnecessary computation or excessive smoothing, or deblocking filtering can still be performed for visual quality purposes.

[0113] When neural network-based model filtering is signaled to be turned on at the CTU level or frame level, some other in-loop filters, such as ALF, Cross Component Adaptive Loop Filter (CCALF), and SAO, may be turned off.

[0114] When neural network-based model filtering is signaled to be turned on at the CTU level or frame level, other in-loop filters may be selectively turned on or off at the CTU level or frame level. For example, if an intra-frame or intra-frame CTU is enabled for neural network-based model filtering, other in-loop filters, such as deblocking filtering, ALF, CCALF, and / or SAO, for the current intra-frame or current intra-frame CTU are disabled.

[0115] [Deblocking filter-assisted neural network-based model filtering] The main purpose of a deblocking filter is to remove blocking artifacts. In a typical situation, the samples filtered / refined by the deblocking filter are most likely located at split or predicted block boundaries. If sample values ​​change before and after the deblocking filter, a deblocking strength map can be generated, where unchanged sample values ​​have an intensity value of zero and changed sample values ​​have a non-zero intensity value. Each non-zero value is determined by the deblocking strength and the dynamic range defined for the intensity map.

[0116] When this deblocking strength map is generated and input to a neural network, it contains prediction information and segmentation information for each image. That is, when this deblocking strength map is input to a neural network, it is not necessary to input the prediction information or segmentation information for each image to the neural network as input to the neural network. This is because only samples located on prediction block boundaries and / or segmentation block boundaries can have non-zero values ​​at their corresponding coordinates on the deblocking strength map.

[0117] 18 is a block diagram illustrating an apparatus for image filtering in video coding using neural networks according to some implementations of the present disclosure. The apparatus 1800 may be a terminal such as a mobile phone, a tablet computer, a digital broadcast terminal, a tablet device, or a personal digital assistant.

[0118] As shown in FIG. 18 , device 1800 may include one or more of the following components: a processing component 1802, a memory 1804, a power component 1806, a multimedia component 1808, an audio component 1810, an input / output (I / O) interface 1812, a sensor component 1814, and a communication component 1816.

[0119] The processing component 1802 typically controls the overall operation of the device 1800, such as operations related to the display, phone calls, data communications, camera operation, and recording operations. The processing component 1802 may include one or more processors 1820 for executing instructions to complete all or some of the steps of the methods described above. Additionally, the processing component 1802 may include one or more modules for facilitating interaction between the processing component 1802 and other components. For example, the processing component 1802 may include a multimedia module for facilitating interaction between the multimedia component 1808 and the processing component 1802.

[0120] Memory 1804 is configured to store different types of data to support the operation of device 1800. Examples of such data include instructions for any applications or methods operating on device 1800, contact data, phone book data, messages, photos, videos, etc. Memory 1804 may be static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or compact disk.

[0121] Power supply components 1806 provide power to the different components of device 1800. Power supply components 1806 may include a power management system, one or more power sources, and other components associated with the generation, management, and distribution of power for device 1800.

[0122] The multimedia component 1808 includes a screen that provides an output interface between the device 1800 and a user. For example, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen that receives input signals from a user. The touch panel may include one or more touch sensors for sensing touches, slides, and gestures on the touch panel. The touch sensors may detect not only the boundaries of a touch or slide motion but also the duration and pressure associated with the touch or slide motion. In some embodiments, the multimedia component 1808 may include a front camera and / or a rear camera. When the device 1800 is in an operating mode, such as a photo mode or a video mode, the front camera and / or the rear camera can receive external multimedia data.

[0123] The audio component 1810 is configured to output and / or input audio signals. For example, the audio component 1810 includes a microphone (MIC). The microphone is configured to receive external audio signals when the device 1800 is in an operation mode such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 1804 or transmitted via the communication component 1816. In some embodiments, the audio component 1810 further includes a speaker for outputting audio signals.

[0124] The I / O interface 1812 provides an interface between the processing component 1802 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0125] The sensor component 1814 includes one or more sensors for providing status assessments in different aspects for the device 1800. For example, the sensor component 1814 can detect the on / off state of the device 1800 and the associated location of the component. For example, the component can be the display and keypad of the device 1800. The sensor component 1814 may also detect changes in the position of the device 1800 or a component of the device 1800, whether or not a user is touching the device 1800, the orientation or acceleration / deceleration of the device 1800, and temperature changes of the device 1800. The sensor component 1814 may include a proximity sensor configured to detect the presence of a nearby object without physical contact. The sensor component 1814 may further include an optical sensor, such as a CMOS or CCD image sensor used in imaging applications. In some embodiments, the sensor component 1814 may further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0126] The communication component 1816 is configured to facilitate wired or wireless communication between the device 1800 and other devices. The device 1800 may access a wireless network based on a communication standard such as Wi-Fi, 4G, or a combination thereof. In one embodiment, the communication component 1816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one embodiment, the communication component 1816 may further include a near-field communication (NFC) module for facilitating short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0127] In one embodiment, apparatus 1800 may be implemented by one or more of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic elements to perform the above-described methods. The non-transitory computer-readable storage medium may be, for example, a hard disk drive (HDD), a solid-state drive (SSD), a flash memory, a hybrid drive or a solid-state hybrid drive (SSHD), a read-only memory (ROM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, etc.

[0128] FIG. 19 is a flowchart illustrating a process for image filtering in video coding using neural networks, according to some implementations of the present disclosure.

[0129] In step 1901, the processor 1820 loads multiple QpMaps into a neural network in one or more QpMap channels, which are each combined with the corresponding YUV channel after three convolution blocks or layers before the concatenation block, as shown in Figure 11.

[0130] For example, the QpMap may have three channels for YUV, namely, QP-Y, QP-U, and QP-V, respectively, because YUV may have different QPs.

[0131] In step 1902, the processor 1820 obtains a QP scaling factor by adjusting a plurality of input QP values ​​associated with the input frame.

[0132] In some embodiments, the encoder may obtain a scaled QP step size by performing subtraction or addition on the corresponding QP input value, and obtain a QP scaling factor based on the scaled QP step size.

[0133] In step 1903, the processor 1820 adjusts a plurality of QpMap values ​​for the neural network according to the QP scaling factor to train and filter the input frame into the neural network.

[0134] In some embodiments, processor 1820 may obtain a QP offset based on a QP offset step size, a lower bound, and an offset index, where the QP offset step size may be a step size for adjusting each input QP value, the lower bound may be an integer value determining the maximum QP value reduction, and the offset index may be a signaled index value. Further, processor 1820 may subtract the QP offset from the QP input value.

[0135] In some embodiments, the encoder may signal a QP offset step size, a lower bound, and an offset index, where the offset index may be an integer between 0 and 3, and the QP offset step size and lower bound may each be a predefined constant value.

[0136] In some examples, the encoder may predefine the QP offset step size and the lower limit. The encoder may further signal an offset index, which may be an integer between 0 and 3. For example, the QP offset step size and the lower limit may be predefined as constant values. In this case, the QP offset step size and the lower limit do not need to be signaled by the encoder. In response to determining that the QP offset step size and the lower limit are not constant values, the encoder signals the QP offset step size and the lower limit to the decoder.

[0137] In some embodiments, the processor 1820 may load multiple input patches associated with an input frame into the neural network in multiple input channels, e.g., YUV channels, and scale the multiple input patches using element-wise multiplication based on the adjusted multiple QP maps.

[0138] In some embodiments, the processor 1820 may align the resolution of multiple input patches across multiple input channels, such as YUV channels, and a concatenated input may be obtained by concatenating the multiple input patches with multiple QP map values ​​via a concatenation block within the neural network, and following the concatenation block, the concatenated input may be loaded into a rest network, which may include multiple rest blocks as shown in FIG. 17, each loading the multiple QP map values ​​as sample value scaling factors that are input to one or more rest blocks within the rest network.

[0139] In some embodiments, the processor 1820 may scale the input to the residual block based on the sample value scaling factor before loading the input into the residual block, for example, the input of the residual block as in FIG. 7A.

[0140] In some embodiments, the processor 1820 may scale the output of the residual block based on the sample value scaling factor, for example, the output of the residual block of FIG. 7B.

[0141] In some embodiments, before obtaining the concatenated input, the processor 1820 may scale the input patches by element-wise multiplication based on the QP map values.

[0142] FIG. 20 is a flowchart illustrating a process for image filtering in video coding using neural networks, according to some implementations of the present disclosure.

[0143] In step 2001, the processor 1820 generates a deblocking strength map indicating prediction or split block boundaries using a deblocking filter.

[0144] In some embodiments, the processor 1820 may generate a deblocking strength map by comparing sample value changes before and after the deblocking filter, where the deblocking strength map may include a plurality of strength values, where a strength value equal to zero may indicate that the corresponding sample value is unchanged and a strength value equal to non-zero may indicate that the corresponding sample value is changed.

[0145] In some embodiments, intensity values ​​that are not equal to zero may be within the dynamic range defined for the deblocking intensity map.

[0146] In step 2002, the processor 1820 inputs the deblocking strength map into a neural network to train and filter the input frame into the neural network.

[0147] In some embodiments, the processor 1820 may input the deblocking strength map to a neural network to replace the input of a prediction map or segmentation map associated with the input frame.

[0148] In some embodiments, in response to determining the coding scenario of the input frame, the processor 1820 may determine whether to perform deblocking filtering on the input frame before inputting the deblocking strength map to the neural network.

[0149] In some embodiments, the processor 1820 may perform deblocking filtering on the input frames before inputting the deblocking strength map to the neural network.

[0150] In some embodiments, the processor 1820 may skip deblocking filtering on the input frame before inputting the deblocking strength map to the neural network.

[0151] In some embodiments, the processor 1820 may skip deblocking filtering on the input frame by restoring the predicted sample values ​​after deblocking filtering.

[0152] In some other embodiments, a non-transitory computer-readable storage medium 1804 having instructions stored thereon is provided that, when executed by one or more processors 1820, cause the processors to perform any of the methods described in Figures 19-20 and above.

[0153] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the present disclosure. Many modifications, variations and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

[0154] The examples have been chosen and described to explain the principles of the disclosure and to enable those skilled in the art to understand the disclosure in its various implementations and to best utilize the underlying principles and various implementations with various modifications as may be suitable for the particular use contemplated. Accordingly, it should be understood that the scope of the disclosure is not limited to the particular examples of implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the disclosure.

Claims

1. 1. A method for image filtering in video coding, comprising: generating a deblocking strength map indicating prediction or partition block boundaries; inputting the deblocking strength map into a neural network to learn and filter input frames through the neural network; A method comprising:

2. generating the deblocking strength map by comparing sample value changes before and after a deblocking filter; The method of claim 1 , wherein the deblocking strength map comprises a plurality of strength values ​​that indicate whether corresponding sample values ​​change.

3. The method of claim 2 , wherein the intensity values ​​that are non-zero are within a dynamic range defined for the deblocking intensity map.

4. The method of claim 1 , further comprising inputting the deblocking strength map into the neural network to replace input of a prediction map or segmentation map associated with the input frame.

5. 2. The method of claim 1, further comprising: determining, in response to determining a coding scenario for the input frame, whether to perform deblocking filtering on the input frame before inputting the deblocking strength map to the neural network.

6. performing the deblocking filtering on the input frames before inputting the deblocking strength map to the neural network; or skipping the deblocking filtering for the input frame before inputting the deblocking strength map to the neural network; or skipping the deblocking filtering on the input frame by restoring predicted sample values ​​after the deblocking filtering; 6. The method of claim 5, comprising performing any one of the following:

7. one or more processors; a memory configured to store instructions executable by the one or more processors; The one or more processors, upon execution of the instructions, generating a deblocking strength map indicating prediction or split block boundaries; An apparatus for image filtering in video coding using a neural network, configured to input the deblocking strength map to a neural network to learn and filter input frames to the neural network.

8. the one or more processors: generating the deblocking strength map by comparing sample value changes before and after a deblocking filter; The apparatus of claim 7 , wherein the deblocking strength map is further configured to comprise a plurality of strength values ​​that indicate whether corresponding sample values ​​change.

9. The apparatus of claim 8 , wherein the intensity values ​​that are non-zero are within a dynamic range defined for the deblocking intensity map.

10. the one or more processors: The apparatus of claim 7 , further configured to input the deblocking strength map into the neural network to replace input of a prediction map or a segmentation map associated with the input frame.

11. the one or more processors:

8. The apparatus of claim 7, further configured to determine, in response to determining a coding scenario for the input frame, whether to perform deblocking filtering on the input frame before inputting the deblocking strength map to the neural network.

12. performing the deblocking filtering on the input frames before inputting the deblocking strength map to the neural network; or skipping the deblocking filtering for the input frame before inputting the deblocking strength map to the neural network; or skipping the deblocking filtering on the input frame by restoring predicted sample values ​​after the deblocking filtering; The apparatus of claim 11 , further configured to:

13. Executing the method for image filtering in video coding according to any one of claims 1 to 6 to generate a bitstream; storing the bitstream; A method for storing a bitstream, including:

14. 1. A computer program product comprising one or more programs executed by a computing device having one or more processors, A computer program product, the program being configured, when executed by the one or more processors, to cause the computing device to perform the method of any one of claims 1 to 6.

15. A computer-readable storage medium storing a bitstream to be encoded by the method for image filtering in video coding according to any one of claims 1 to 6.

Images