Loop Filtering Using Neural Networks

Abstract

Methods, systems, bitstream syntax, and fixed-point implementations for loop filtering using neural networks in image and video processing are described. Given an input image, a hybrid luma-chroma filter is proposed, where the luma and chroma components are first processed by a first neural network, and the output of the first network is then processed by separate luma and chroma sub-networks. Finally, the outputs of the separate luma and chroma sub-networks are concatenated to produce a filtered output of the input image. An efficient method using CP decomposition is also described. A method for indicating the position of the neural net loop filter relative to other filters such as the adaptive loop filter (ALF) is also discussed.

Description

Technical Field

[0001] Cross - References to Related Applications This application claims priority to Indian Provisional Patent Application No. 202241038279, filed on July 4, 2022; Indian Provisional Patent Application No. 202241074543, filed on December 22, 2022; Indian Provisional Patent Application No. 202341017121, filed on March 14, 2022; and U.S. Provisional Patent Application No. 63 / 432,613, filed on December 14, 2022.

[0002] Technique This document generally relates to images. More specifically, certain embodiments of the present invention relate to filtering images using a neural network.

Background Art

[0003] In 2020, the MPEG group of the International Organization for Standardization (ISO), in collaboration with the International Telecommunication Union (ITU), released the first version of the Versatile Video Coding (VVC) standard, also known as H.266 (Reference [8]). More recently, the same joint group (JVET) and experts in still - image compression (JPEG) have started working on the development of a next - generation coding standard that provides improved coding performance over existing image and video coding techniques. As part of this research, coding techniques based on artificial intelligence and deep learning are also being considered. As used herein, the term "deep learning" refers to a neural network having at least three layers, preferably more than three layers.

[0004] As understood by the inventors, improved techniques for encoding images and videos based on neural networks are described herein.

[0005] The approach described in this section is an approach that could have been pursued, but is not necessarily an approach that has been previously conceived or pursued. Thus, unless otherwise noted, none of the approaches described in this section should be assumed to be eligible as prior art solely by virtue of being included in this section. Similarly, unless otherwise noted, problems identified with one or more approaches should not be assumed to have been recognized in any prior art based on this section.

Brief Description of the Drawings

[0006] Certain embodiments of the present invention are shown by way of example and not limitation in the figures of the accompanying drawings. Like reference numerals refer to like elements.

[0007]

Figure 1

[0008]

Figure 2

[0009]

Figure 3

[0010]

Figure 4A

[0011]

Figure 4B

[0012]

Figure 4C

[0013]

Figure 4D

[0014]

Figure 4E

[0015]

Figure 4F

[0016]

Figure 5A

[0017]

Figure 5B

[0018]

Figure 6

[0019]

Figure 7A

[0020]

Figure 7B

[0021]

Figure 7C

[0022] Exemplary embodiments for loop filtering using neural networks in image and video coding are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments of the invention. It will be apparent, however, that the various embodiments of the invention may be practiced without these specific details. On the other hand, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obfuscating, or confusing the embodiments of the invention.

[0023] Summary The exemplary embodiments described herein relate to image and video coding using neural networks. In one embodiment, a processor receives an input image having luma and chroma color components; applies a first neural network (NN) to both the luma color component and the chroma color component of the input image to generate a first luma output and a first chroma output; applies a second neural network to the first luma output to generate a second luma output; applies a third neural network to the first chroma output to generate a second chroma output; concatenates the second luma output and the second chroma output to generate a colored filtered output.

[0024] In a second embodiment, the processor receives a high-level syntax indicating that neural network-based loop filtering (NNLF) is enabled for decoding the current picture. The processor then: parses the high-level syntax to extract parameters indicating a recommended position of an adaptive loop filter (ALF); decodes the current picture based on the ALF position parameter to generate an output picture, where the ALF position parameter specifies one of: performing ALF filtering after the NNLF; performing ALF filtering before the NNLF; replacing the ALF filtering by a convolutional neural network (CNN) located after the NNLF; and performing ALF filtering before the enhanced NNLF using ALF classification data as the NNLF input.

[0025] Exemplary Coding Model Using Deep Learning ​Deep learning-based image and video compression techniques are becoming increasingly popular and are an active area of research. Current research in neural network (NN)-based coding can be divided into two general frameworks. Namely, a "hybrid" neural network-based framework (e.g., reference [1]) that simply replaces one or more existing encoding or decoding modules with their corresponding neural network-based implementations, where each NN module is trained and optimized on its own, and an "end-to-end" neural network (reference [9]) where training and optimization are performed on the entire network. The proposed neural net filter is applicable to any architecture. The term YUV420 represents the luma-chroma color space, where the chroma is subsampled by a factor of 2 in both the horizontal and vertical dimensions, such as YCbCr 4:2:0. An example refers to the neural network loop filter (NNLF), but this technique can be applied to a variety of other NN-based filters that remove noise artifacts or improve image quality, such as NN post-filtering, super-resolution filtering, etc.

[0026] Existing NNLF designs (references [2]-[4]) provide an improvement of nearly 5% - 10% in coding efficiency (BDrate), but at a very high computational cost in the range of 33 - 625 kMAC (thousands of multiply-accumulations) operations per pixel. The embodiments presented in this specification provide improved coding efficiency with reduced computational cost.

[0027] Hybrid luma-chroma NN model In the current literature, the NNLF model can be divided into two main categories, namely the combined luma / chroma model and the separate luma / chroma model. These categories are based on whether the luminance (Y) signal and the chroma (UV) signal share the same NN. For the combined model, two typical approaches are used to supply the YUV420 signal to the NN. In Reference 3, as shown in Figure 1, the Y signal is interleaved with the UV signal, and a total of six planes are used. In addition to these six planes, four additional planes of the Y, U, V boundary strength maps (BS Y BS U BS V ) and the slice quantization parameter (QStep) are used to form ten input channels as shown in Figure 1. As shown in Figure 1, this architecture includes n filter blocks including n - 2 filter blocks used as hidden layers. The first layer (105) uses a 3×3 convolutional layer with an activation layer (e.g., Leaky ReLU) and outputs K channels (feature maps). Each hidden filter block (110) consists of two 1×1 convolutional layers with an activation layer (e.g., Leaky ReLU) in between, followed by a 3×3×M×K convolutional layer, where the value of M is set to be larger than the value of K, and the notation C×C×M×K indicates a C×C convolutional layer with M inputs and K outputs. By setting different values for n, M, and K, models with different complexities can be created (e.g., n = 13, M = 72, K = 24, or n = 14, M = 216, K = 72). The final layer outputs a filtered luma-chroma residual channel (L = 6) consisting of four luma planes (Y signal) and two chroma planes (UV signal). These are added back to the NNLF input sample to obtain the final filtered output, which is used as the input to the adaptive loop filter (ALF) in Reference [3].

[0028] The input size of the NN filtering process is 144×144, including the current coding tree unit (CTU) and eight samples adjacent to each side of the current CTU. Luma samples are interleaved into four 72×72-sized blocks before being used as input to the filtering process. The output tensor corresponds to the filtered CTU samples organized as 64×64 blocks by cropping the central region of the 72×72 filtered output that includes four luma blocks and / or two chroma blocks. The output tensor of the NNLF corresponds to the filtered luma-chroma residual channels, which are scaled and added back to the NNLF input samples to obtain the final filtered output in reference [3]. During the training stage, scaling is not applied, and the mean squared error (MSE) between the filtered output and the original samples is minimized using a weighted L2 loss function. During the inference stage, the scaling parameters for luma and chroma are signaled in the slice header by the encoder to minimize the mean squared error (MSE) between the filtered samples and the original samples of the current frame or image.

[0029] Alternatively, as described in reference [4], chroma (UV) samples are first upsampled to match the resolution of the Y component, and all three components are fed into a single NN filter. At the output of the NN filter, the chroma components are downsampled to their original (input) resolution (e.g., the resolution of the 4:2:0 format).

[0030] In a separate luma / chroma NN filter as shown in FIG. 2 (reference [2]), the Y signal and the UV (CbCr) signals are filtered using two different, separate NNs to generate the filtered output (Y'Cb'Cr'). In some embodiments, the chroma NN can also use an auxiliary luma input (AuxY).

[0031] Figure 3 shows an exemplary split architecture for luma / chroma filtering according to an embodiment. The input to the network is the combined Y and UV signals. For the NN, part of the network is shared by both luma and chroma, and part of the network is split into two separate paths for the luma and chroma components. The corresponding luma / chroma channels can also be different. The motivation behind this is that the characteristics of the luma and chroma signals are different. Thus, the chroma component may require fewer channels and / or fewer layers than the luma component without losing performance. This is why separate paths can achieve similar performance while reducing complexity. Further, since the luma / chroma signals still have some inter-component correlation, the combined path at the beginning or end of the network can help utilize such correlation.

[0032] As an example, in Figure 3, the architecture combines the following NN components.

[0033] Input N×N, Y×4+U+V, BS information (x3), QStep (e.g., 72×72×10).

[0034] Combined input path: · (3×3) convolution, 3×3×10×M (10 inputs, M outputs) · Leaky ReLU (Common hidden layer) · Conv, 1×1×M×M · Leaky ReLU · Conv 1×1×M×K · Conv 3×3×K×K

[0035] Luma path, n y number of hidden layers: · Conv, 1×1×Ky×My · Leaky ReLU · Conv 1×1×My×Ky · Conv 3×3 ×Ky×Ky Followed by Conv, 3×3×Ky×Ly

[0036] Chrominance path, n c hidden layers: ·Conv, 1×1×Kc×Mc ·Leaky ReLU ·Conv 1×1×Mc×Kc ·Conv 3×3×Kc×Kc Followed by Conv, 3×3×Kc×Lc

[0037] Sum output path ·Channel concatenation Output: N'×N', Y'×4 + U' + V' (e.g., 64×64×6)

[0038] Compared with FIG. 1, the following differences are also worth noting. In the first layer (305), instead of using a 3×3×10×K convolutional layer (see 105), a 3×3×10×M convolutional layer is used, and in the common hidden layer (310), instead of using a 1×1×K×M convolutional layer (see 110), a 1×1×M×M convolutional layer is used.

[0039] In one embodiment, for the n y hidden layers in the luminance path, and the n c hidden layers in the chrominance path, the complexity in terms of kMAC / pixel of the proposed architecture can be calculated as follows. ·Total MAC in the luminance path = n y *(2*K y *M y +9*K y *K y ) + 9*K y *L y ·Total MAC in the chrominance path = n c *(2*K c *M c +9*K c *K c ) + 9*K c *L c · Total MAC in the common path = 9 * 10 * M + M * M + K * M + 9 * K * K For example · When N = 72, K = 24, and M = 72 in the common path, the number of MACs = 18,576 · In the luma path, n y = 10, K y = 16, M y = 48, L y = 4, the number of MACs = 38,976 · In the chroma path, n c = 10, K c = 8, M c = 24, L c = 2, the number of MACs = 9,744 The total MAC for processing 4 luma (and 2 chroma) samples is 67,296, and the boundary pixels from 72×72 are ignored in the output to obtain 64×64. Therefore, the MACs per pixel = (67,296 * 72 * 72) / (64 * 64 * 4) = 21.292 kMACs / pixel. Compared with the reference [3] reporting a complexity of about 33.6 kMACs / pixel, the computational load required by the proposed architecture is about 0.633.

[0040] Tensor decomposition method Applying tensor decomposition is one of many methods for reducing the complexity of NN calculations. There are several tensor decomposition methods. For example, the depth-wise separable convolution (DSC) method has been tried in some JVET submissions such as reference [5]. In one embodiment, the complexity of the NN is reduced using CP decomposition (reference [6]). In CP decomposition (CP is derived from the CANDECOMP / PARAFAC model), a 4D convolution kernel tensor is decomposed into a sequence of 4 convolutional layers with small kernels. The first convolutional layer is a point-wise convolution, the second and third layers are spatial convolutions in the X and Y directions, and the fourth convolution is again a point-wise convolution in the channel dimension.

[0041] The ordinary convolution for output channel t can be written as follows.

Equation

Equation

Equation

Equation

[0042] As an example, Figure 4A shows an example of applying CP decomposition (420) to the architecture shown in Figure 1 modified in the common layer of Figure 3 (that is, in the first layer, instead of using a 3×3×10×K convolutional layer (see 105), a 3×3×10×M convolutional layer is used, and in the first hidden layer, instead of using a 1×1×K×M convolutional layer (see 110), a 1×1×M×M convolutional layer is used.).

[0043] As shown in Figure 4A (details of 420), the 3×3 convolution of each hidden layer is decomposed into four layers with rank R (see 420). · First layer: Pointwise convolution of 1×1×K×R · Second layer: Separable convolution of 3×1×R×R · Third layer: Separable convolution of 1×3×R×R · Fourth layer: Pointwise convolution of 1×1×R×K

[0044] Applying CP decomposition, for R = 24 applied to all hidden layers, the complexity reduction factor is about 1.6 times compared to reference [3]. When using R = 12, the complexity reduction factor is about 1.8 times.

[0045] As shown in Figures 4A and 4B, in CP decomposition (420), the first decomposed layer (415) (1×1×K×R) can be fused with the previous four layers (405) of 1×1×M×K of the hidden units, resulting in an effective fused single layer (430) (1×1×M×R). Similarly, the last CP decomposition layer (425) (1×1×R×K) can be fused with the first layer (410) (1×1×K×M) of the next hidden units to achieve the same function by a single fused (1×1×R×M) layer (440). This fusion of adjacent 1×1 pointwise convolutions is shown in Figure 4B. This fusion is possible because there is no non - linear activation between the layers to be fused, resulting in further complexity reduction. The fused network in Figure 4B may introduce a slight difference in the inference output due to the difference in the order of floating - point operations, but has a negligible difference in the encoding performance compared to the non - fused CP - decomposed NNLF filter in Figure 4A.

[0046] Applying CP decomposition with fusion, for R = 24 applied to all hidden layers, the complexity reduction factor is about 2.1 times compared to reference [3]. When using R = 12, the complexity reduction factor is about 3.3 times.

[0047] Figure 4C shows the network architecture of Figure 4B with additional CP decomposition of the input 3×3 convolutional layer (420A) and the output 3×3 convolutional layer (420B). The input 3×3×10×M convolutional layer (420A) is decomposed into four layers of pointwise convolution of 1×1×10×R, separable convolution of 3×1×R×R, separable convolution of 1×3×R×R, and pointwise convolution of 1×1×R×M. Similarly, the output 3×3×K×L convolution (420B) is decomposed into four layers of pointwise convolution of 1×1×K×R, separable convolution of 3×1×R×R, separable convolution of 1×3×R×R, and pointwise convolution of 1×1×K×L. For R = 24, due to the additional decomposition of the input and output layers, the overall complexity reduction factor is now about 2.3 times.

[0048] Figure 4D shows an exemplary embodiment for applying CP decomposition to the split architecture of Figure 3. As described above, the layers of CP decomposition (420) can also be fused with the preceding or succeeding 1×1 convolutional layer as appropriate. CP decomposition with rank R = 24 applied to all hidden layers achieves a complexity reduction factor of 2.5 times. CP decomposition with rank 24 applied to all hidden layers and the fusion of adjacent 1×1 pointwise convolutions achieves a complexity reduction factor of 3 times.

[0049] Figure 4E depicts the network architecture of Figure 4D with additional CP decompositions of the input 3×3 convolutional layer (420A) and the lumina and chroma output 3×3 convolutional layers (420D, 420C). The input 3×3×10×M convolutional layer (420A) is decomposed into four layers of 1×1×10×R pointwise convolution, 3×1×R×R separable convolution, 1×3×R×R separable convolution, and 1×1×R×M pointwise convolution. The output lumina 3×3×Ky×Ly convolution (420D) is decomposed into four layers of 1×1×Ky×Ry pointwise convolution, 3×1×Ry×Ry separable convolution, 1×3×Ry×Ry separable convolution, and 1×1×Ky×Ly pointwise convolution. Similarly, the output chroma 3×3×Kc×Lc convolution (420C) is decomposed into four layers of 1×1×Kc×Rc pointwise convolution, 3×1×Rc×Rc separable convolution, 1×3×Rc×Rc separable convolution, and 1×1×Kc×Lc pointwise convolution. For R = 24, Ry = 16, and Rc = 8, using the additional decomposition of the input and output layers, the overall complexity reduction factor is approximately 3.4 times.

[0050] As shown in FIG. 4F, the decomposed separable convolutional layers of 3×1 (417A) and 1×3 (417B) shown in FIG. 4B can be fused into a single 3×3 separable convolution (445). This fusion helps to reduce the number of layers in the network, which can help to reduce the latency and memory access overhead of the network, and can utilize a hardware / software implementation that is optimized for a 3×3 convolution kernel size but not for 3×1 or 1×3 row-wise or column-wise convolution operations. This fusion is possible because there is no non-linear activation between the layers being fused and the fused block is mathematically equivalent to the layers being fused. The additional "mac" (multiply-accumulate) operations due to the fusion (9×R×R in the fused layer versus 6×R×R before fusion) have a negligible impact on the overall complexity of the filtering. The resulting increase in "mac complexity" in FIG. 4F is less than 2% of the increase in FIG. 4B, while an implementation-specific speed increase of up to 30% was observed.

[0051] NNLF and ALF Pipeline Issues When the NNLF is used in the VVC framework (Reference [8]), the position of the NNLF is very important. VVC includes several in-loop filters such as a deblocking filter (DF), sample adaptive offset (SAO), adaptive loop filter (ALF), and component - to - component ALF (CCALF) after inverse luma mapping - chroma scaling (LMCS), which are applied to improve the quality of the decoded signal. When the NNLF is proposed, the DF / SAO can be incorporated into the NNLF. So far, all JVET submissions have not been able to eliminate the ALF and CCALF due to the additional gains brought by the ALF and CCALF. In addition, since the ALF and CCALF need to calculate coefficients based on the distortion between the decoded / filtered signal and the original signal, it has been found that the best gain can be achieved by placing the ALF and CCALF after the NNLF. However, in practical applications, the NNLF is most likely to be implemented by a graphics processing unit (GPU), while the ALF, CCALF, and other decoding modules are implemented by the GPU, so the placement of the ALF filter and CCALF filter may cause pipeline problems. To solve such pipeline problems, the following embodiments are described.

[0052] First Embodiment: Replace the ALF and CCALF with a convolutional neural network (CNN). The tests show that the main gains from ALF and CCALF are for the chroma components. Conceptually, the CNN is essentially ALF and CCALF. In certain embodiments, a separate CNN can be added after NNLF. More specifically, the separate CNN can include only layers that perform a convolution or dot product of the convolutional kernel and the input matrix of the layer. This CNN can be pre-trained or can be trained online (so-called CNN model update). For the latter method, it is necessary to transmit the coefficients together with the bitstream. The updated coefficients can be signaled for intra-picture or inter-picture references by other pictures.

[0053] Second Embodiment: Enhanced NNLF An alternative method is to use the NNLF filter as the final stage of all loop filters, but with additional input ALF classification data. This architecture is called "enhanced NNLF". The advantage of this approach is that it does not affect the hardware pipeline of the conventional loop filters in VVC (e.g., deblocking, SAO, and ALF). This method uses a 4×4 block classification metric based on horizontal, vertical, diagonal gradients, and local activity, which are used in the current VVC ALF filter as additional feature inputs to the NNLF filter. In addition, separate NNLF models for luma and chroma can be used.

[0054] FIG. 5A shows an embodiment for VVC loop filtering using a combined (or hybrid) NNLF network. As shown in FIG. 5A, the reconstructed luma sample (RecY) is sequentially processed by inverse luma mapping (inverse LMCS), deblocking, SAO, and ALF filtering to generate luma ALF classification data (502) and luma ALF samples (504). The output of luma SAO is also processed by the Cb CCALF filter and the Cr CCALF filter.

[0055] The reconstructed chroma samples (RecCb, RecCr) are sequentially processed by deblocking, SAO, and ALF, and their outputs are added to the corresponding outputs of the Cb / Cr CCALF filters to generate Cb ALF samples (506) and Cr ALF samples (508). Finally, the ALF classification data (502), the luma ALF samples (504), and the chroma AL samples (506, 508) are merged together as inputs to the NNLF, along with the BS information and the QStep data, to generate the filtered output Y'Cb'Cr'.

[0056] FIG. 5B shows an alternative embodiment for VVC loop filtering using separate luma NNLF network and chroma NNLF network. The front part of the design is the same as that in FIG. 5A, but the generated data (e.g., 502, 504, 506, and 508) are fed to two separate networks. As shown in FIG. 5B, the luma NNLF filter does not use any chroma samples, but the chroma NNLF filter can use the filtered luma ALF samples (504) as part of its input.

[0057] Syntax Example The proposed tool can be communicated from the encoder to the decoder using high-level syntax (HLS) that can be part of a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), slice header (SH), or as part of supplementary metadata such as supplementary enhancement information (SEI) data. An exemplary syntax is shown in Table 1. Alternatively, such signaling may not be required if a specific architecture is predetermined and known to both the encoder and the decoder.

Table 1

Table 2

Table 3

[0058] Fixed-Point Implementation Embodiments of the present disclosure present various methods for quantizing the floating-point convolutional weights and bias layers of a neural network to achieve an efficient fixed-point neural network without affecting its performance with respect to the accuracy of the output. The main goals of the fixed-point integer implementation of any neural network are as follows. 1. Bit-accurate inference: Achieve the same output on any hardware or software platform for a given input. 2. Low complexity: Implementations that are friendly to software and hardware using integer arithmetic operations aimed at reducing complexity, higher throughput, and power efficiency. 3. Accuracy: Maintain the highest possible accuracy, achieving the smallest possible deviation compared to floating-point inference.

[0059] All methods in the present disclosure can be applied to any convolutional feed-forward neural network, but the examples and implementations are realized using the above-described low-complexity neural network-based loop filter (NNLF).

[0060] Loop filtering is typically a canonical process in any video decoder, so it is highly desirable for the decoder to have accurate output bits on any hardware / software platform. Since accurate output bits are not achievable with floating-point inference, a fixed-point implementation is required. For example, an NNLF floating-point model is initially trained in either PyTorch or TensorFlow and later converted to fixed-point for this process.

[0061] All methods proposed in the present invention aim to implement the integer implementation of NNLF. Without limitation, as an example, the integer implementation and verification are performed using SADL (Small Ad hoc Deep-Learning Library), a lightweight library for performing neural network inference in pure C++. Further, typically, a 16-bit fixed-point integer implementation (int16) is more desirable than a 32-bit integer implementation (int32) for faster implementation and lower memory cost. Without limitation, as an example, the performance of the fixed-point implementation is measured based on how it performs compared to its floating-point implementation shown in Table 4 for the test images of the JVET "Class D" set for All Intra, Main 10. For example, compared to a VVC encoder without an NNLF filter, a trained NNLF filter in a floating-point implementation improves Y-PSNR by 5.04%.

Table 4

[0062] Figure 6 shows an example of a fixed-point representation of a number. The K bits before the virtual decimal point represent the integer part including the sign bit, and the F bits after the decimal point represent the fractional part. The Q format or Q factor of the fixed-point representation is determined by the number of F bits representing the fractional part. Thus, for example, the term Q14 indicates a fixed-point implementation with F = 14 bits. Thus, for Int16, Q11 indicates that 5 bits are used for the sign and integer part, and 11 bits are used for the fractional part, for a total of 16 bits.

[0063] The high-precision and efficient fixed-point implementation of NNLF involves determining the optimal Q format for all weights and biases of all convolutional layers, as well as the optimal Q format for the inputs and intermediate outputs of each layer and the final output of the NNLF filter.

[0064] All Q format-related operations in fixed-point convolution are described below. 1. The normalized input x (0,1) is multiplied by the optimal Q factor (e.g., Q in ) based on the required precision. 2. The weights (A) and biases (B) of each convolution are also represented by their own Q factors Q A and Q B respectively. 3. The resulting convolution output y = Ax + B is also preferably in the Q in format. Thus, the multiplication and accumulation result of y' = Ax is right-shifted by only the Q factor of the conv layer Q A to return the output y' to the input Q format Q in . 4. In the bias addition layer y = y' + B, the input y' is scaled up or down based on the difference between the input and bias q formats (Q B - Q in ) so as to match the bias Q format Q B before addition. The Q format of the bias layer Q B and the input Q inIt is desirable for them to be identical so that there is no loss of precision or data overflow due to the additional introduced shift. All remaining sections show implementation methods and corresponding PSNR results for int32 and int16 fixed-point implementations.

[0065] Table 5 shows some experimental results using various fixed-point implementations and how they compare to the floating-point results in Table 4. Term (w) refers to the weight and term (b) refers to the bias.

[0066] Using Int32, Q14, the results of this configuration match well with the reference floating-point results. When Q11 is used, as expected, a loss of performance is detected. Applying dynamic Q for the weight but fixed Q11 for the bias improves the performance; however, when applying the more desirable 16-bit implementation, a significant loss of performance due to variable overflow during the convolution operation is observed.

[0067] The convolution weights and biases of some layers have very high magnitudes. These values are mainly seen in 1×1 convolution layers. These values are very high in the floating-point domain. Thus, even after right-shifting only by the Q factor of that layer, the output of the layer size tends to increase. This means that there are few convolution layers where the output energy is higher than the input. This increase in size causes overflow in subsequent convolution layers.

Table 5

[0068] Int16 implementation based on dynamic weight renormalization In principle, the output of the NNLF filter after all convolutional layers has the same (or lower) energy as the input. When there are few layers in the model with large convolutional weights that cause amplification, there are few other layers in the subsequent pipeline that have much smaller weights that reduce the dynamic range to the dynamic range of the input. Since convolutional layers with high magnitudes cause overflow, in order to prevent overflow, it is desirable to normalize the convolutional weights of such layers (e.g., i) by a scaling factor W (where W > 1), as shown in the following equation.

Number

[0069] Output y i ' is now normalized by W, so y j = y j To ensure ', this factor needs to be rescaled in subsequent (or previous) layers (e.g., j) where the magnitude of the convolutional weights is sufficiently smaller, as shown in the following equation.

Number

Number

[0070] In short, layers with larger magnitudes are divided by the scaling factor, and layers with smaller magnitudes have these scaling factors multiplied back. These normalizations can be applied such that the NNLF floating-point output is mathematically the same before and after normalization.

[0071] In the above description, it should be noted that when the scaling factor W is propagated to a subsequent layer (j > i + 1) that is not immediately after the current layer (i), all intermediate bias layers between "i" and "j" must also be normalized by W. This is specified by the following equation (7).

Equation

[0072] The factor propagated to a subsequent layer (j > i + 1) for rescaling is also called the propagation factor P in Methods 2 and 3 described below.

[0073] Equations (4) to (7) are applicable only when normalization is performed in the forward direction (j > i). When normalization is performed in the reverse direction (j < i), the weights and biases of the current layer (i) and the previous layers are modified as shown in the following equations (8), (9), and (10).

Equation

[0074] Based on this weight normalization principle, the following three different approaches were tested for 16-bit fixed-point inference. 1. Normalization of localized hidden layers. 2. Global normalization based on maximum weights using forward propagation of normalization. 3. Global normalization based on geometric mean and bidirectional propagation of normalization.

[0075] Method 1: Normalization of Localized Hidden Layers This method is shown in Figure 7A and includes the following steps. 1. The second 1×1 convolutional layer (e.g., 705) of each hidden layer is divided by a factor W based on the largest magnitude weights of that layer, such that the resulting magnitude is constrained to be less than a fixed threshold (e.g., threshold = 4). 2. This division is accepted by scaling the weights of the subsequent 3×1 and 1×3 (710, 715) convolutional layers of the same hidden layer by a factor of √W. 3. The re-normalized network is first verified for floating point inference and then int16 fixed point inference is performed using dynamic Q factors for convolutional weights and fixed Q11 for biases. The results of this approach are shown in Table 6. The results of Method 1 show a significant improvement in int16 encoding performance but are still significantly worse than the int32 performance (see Table 5) using dynamic Q and fixed Q11 for biases. The main conclusions and drawbacks of this method are as follows. 1. Local normalization can still lead to overflow. 2. If the scaling factor W is very large due to unusually high value weights, there is insufficient capacity for normalization.

[0076] Method 2: Global normalization based on maximum weights using forward propagation. This method is shown in Figure 7B and includes the following steps. 1. Starting from the first hidden layer, evaluate the maximum weight of each convolutional layer. The initial propagation factor P is set to 1. 2. Determine the division factors (Di1, Di2) for each 1×1 convolutional layer (702, 705) based on the largest magnitude weights in that layer. Apply the division factors such that all 1×1 convolutional weights are constrained to be less than 1 in magnitude. 3. Based on Di1, Di2, and any previous propagation factor P > 1, determine the multiplication factors (Mi1, Mi2) for the subsequent 3×1 and 1×3 convolutional layers (710, 715). The maximum allowable target weights for the 3×1 and 1×3 layers are constrained to be less than a fixed value (e.g., 5). 4. If (Mi1 * Mi2) cannot fully accept (P * Di1 * Di2), the propagation factor P is updated to (Di1 * Di2 * P) / (Mi1 * Mi2) and propagated forward. Otherwise, if it is fully accepted, P is updated to 1. 5. Divide all of the future bias layers by the propagation factor (P) until they are fully considered during the process. 6. Normalization is performed in the forward direction until all 1×1 convolutional layers in the hidden layer are normalized. 7. The propagation factor P remaining after the last 3×1 layer and 1×3 layer cannot be propagated any further, so it is multiplied by the last 1×1 layer in the network. 8. The re-normalized network is first verified for floating-point inference, and then int16 fixed-point inference is performed using dynamic Q factors for convolutional weights and fixed Q11 for biases.

[0077] The results of this approach are also shown in Table 6. Method 2 shows a significant improvement in int16 coding performance compared to Method 1, but its performance is still worse than that of int32 using dynamic Q for weights and Q11 for biases. The main conclusions and drawbacks of this method are as follows. 1. Any constraints on the maximum allowable size for 1×1, 1×3, and 3×1 convolutions. 2. One-way forward propagation may not be sufficient to fully absorb large propagation factors.

[0078] Method 3: Global normalization based on geometric mean and bidirectional propagation. This method aims to find the geometric mean (GM) of the weights of the maximum size of each layer and sequentially and iteratively normalize the network weights so that all weights of the maximum size are exactly scaled to GM. The propagation of weights is performed in both the forward and reverse directions. Given m numbers a1, a2, …, a m When given, their geometric mean is defined as follows.

Number

[0079] This method is shown in Figure 7C and includes the following steps. 1. Calculate the geometric mean (GM) of all maximum-sized weights Amax from each layer (755). i =Max(abs(A i )) 2. Set the initial propagation factor P to P = 1 and GlobalMax = Max(Amax i )(780). 3. Determine the hidden layer "i" with the highest-sized weight and normalize its weight by the scaling factor W such that W = GlobalMax / GM (782). Assuming subsequent forward propagation (784), adjust its bias value according to Equation (4) (782). The propagation factor P is updated to P = P*W. 4. Forward propagation normalization (784): Propagate the scaling factor P forward to the layer where (j>i) and the maximum weight is smaller than GM. The weight of layer "j" where the maximum weight Amax j is smaller than GM is scaled up by the multiplication factor M j =min(P,GM / Amax j ) so that the scaled maximum weight is below GM. The scaling operation is performed according to Equation (6) for layer j and Equation (7) for layer k (i<k<j). The propagation factor P is updated to P = P / M j . Forward propagation normalization continues until P becomes 1 or the last layer is reached. The value of P at the end of forward propagation normalization is denoted as Pb (the backward propagation factor). If Pb = P>1, backward propagation normalization (788) is required. However, before this step, the bias value of the hidden layer i needs to be readjusted as in Equation (12) (786). Here, A i ' and B i ' represent the adjusted values after Step 3 (782) using Equation (4).

Number

[0080] In another embodiment, in step (782), it may be determined to start from the reverse propagation (788). Then, for the hidden layer i, its bias value needs to be adjusted based on Equation (8), and steps (784) and (788) need to be exchanged. Also, the value of P after reverse propagation normalization is denoted as Pf (forward propagation factor). When Pf = P > 1, before the forward propagation normalization starts, the bias value of the hidden layer i needs to be readjusted as in Equation (13). Here, A i ' and B i represent the adjusted values after step 3 (782) using Equation (8). [Number]

[0081] The results of this approach are also shown in Table 6. Method 3 shows a further improvement in int16 encoding performance. The results are only slightly worse than the int32 performance (see Table 5) using dynamic Q for weights and fixed Q11 for bias values. [Table 6]

[0082] To better understand Methods 1 - 3, exemplary values of weights and bias values in certain hidden layers before and after the proposed normalization process are given in the appendix, without limitation. References Each of the references listed in this specification is hereby incorporated by reference in its entirety. The term JVET refers to the Joint Video Experts Team of ITU - T SG16 WP3 and ISO / IEC JTC 1 / SC29. [Non - Patent Document 1] Dong Liu, Yue Li, Jianping Lin, Houqiang Li, Feng Wu, “Deep learning-based video coding: A review and a case study,” https: / / arxiv.org / abs / 1904.12462. (Reference 1)

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 6

Non-Patent Document 7

Non-Patent Document 8

Patent Document 9

[0083] Implementation of an Exemplary Computer System Embodiments of the present invention may be implemented using a computer system, an electronic circuit, and a system, microcontroller, field programmable gate array (FPGA), or other integrated circuit (IC) device such as a configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions regarding loop filtering using a neural network for image and video coding, such as those described herein. The computer and / or IC may calculate any of a variety of parameters or values related to loop filtering using a neural network for image and video coding described herein. Embodiments of images and videos may be implemented in hardware, software, firmware, and various combinations thereof.

[0084] Certain implementations of the present invention include a computer processor that executes software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. may implement a method related to loop filtering using a neural network for image and video coding as described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. The program product can include any non-transitory tangible medium that carries a set of computer-readable signals that, when executed by a data processor, include instructions that cause the data processor to perform the methods of the present invention. The program product according to the present invention may be in any of a wide variety of non-transitory and tangible forms. The program product can include physical media such as, for example, a floppy (registered trademark) disk, a magnetic data storage medium including a hard disk drive, an optical data storage medium including a CD ROM, a DVD, a ROM, an electronic data storage medium including a flash RAM, and the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0085] When a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, the reference to that component (including the reference to "means") shall be construed to include any component that performs the function of the recited component (e.g., is functionally equivalent), including components that are not structurally equivalent to the disclosed structure that performs the function in the illustrated exemplary embodiments of the present invention, as an equivalent of that component.

[0086] Equivalents, Extensions, Alternatives, and Others Exemplary embodiments regarding loop filtering using neural networks for image and video coding are thus described. In the above specification, embodiments of the present invention have been described with reference to a number of specific details that may vary from implementation to implementation. Thus, the only and exclusive indicator of what the present invention is and what is intended by the applicant to be the present invention is the set of claims issued from this application, including any subsequent corrections, in the specific form in which such claims are issued. Any definitions explicitly set forth herein for terms included in such claims shall govern the meaning of such terms as used in the claims. Thus, limitations, elements, characteristics, features, advantages, or attributes not explicitly recited in the claims should not in any way limit the scope of such claims. Thus, this specification and the drawings should be regarded as illustrative rather than restrictive in nature.

[0087] Appendix This appendix provides exemplary numerical results during the normalization process for each of Methods 1, 2, and 3.

[0088] Method 1 Legend: Normalization by W is shown in italics, and scaling by √W is shown in bold. In all steps, W = 8. Hidden layer I is processed in Step 1, and its coefficients remain the same for all subsequent steps. Hidden layer I+1 is processed in Step 2, and its coefficients remain the same for all subsequent steps. Hidden layer I+2 is processed in Step 3, and its coefficients remain the same for all subsequent steps. [Table 7]

[0089] Method 2 Legend: Normalization by Di1 and Di2 is shown in italics, and scaling by Mi1 and Mi2 is shown in bold. The hidden layer i (i = 1, 2, …, n) is processed in step i, and its coefficients remain the same for all subsequent steps.

Table 8

[0090] Method 3 GM = 1.8981781. Steps 1 to 3 are exemplary forward propagation normalization steps.

Table 9

Claims

1. A method for performing image filtering using a neural network, the method comprising: accessing an input image having a luminance color component and a chroma color component; applying a first neural network (NN) to both the luminance color component and the chroma color component of the input image to generate a first luminance output and a first chroma output; applying a second neural network to the first luminance output to generate a second luminance output; applying a third neural network to the first chroma output to generate a second chroma output; concatenating the second luminance output and the second chroma output to generate a colored filtered output. Method.

2. The first neural network is: a 3×3 convolutional (CONV) network (3×3×S×M) having S inputs and M outputs, followed by a non-linear activation (NLA) block, followed by a 1×1×M×M CONV network and a second NLA block, followed by a 1×1×M×K convolutional network and a 3×3×K×K convolutional network to generate the first luminance output having Ky signals and the first chroma output having Kc signals, where K = Ky + Kc. The method according to claim 1.

3. The second neural network is: an input of Ky luminance signals, one or more luminance hidden layer blocks for generating the second luminance output, followed by a 3×3×Ky×Ly convolutional block. The method according to claim 1.

4. One of the one or more luminance hidden layer blocks is: a 1×1×Ky×My CONV network, followed by a non-linear activation block, followed by a 1×1×Ky×Ky CONV network, followed by a 3×3×Ky×Ky CONV network. The method according to claim 3.

5. The third neural network is: an input of Kc chroma signals, one or more chroma hidden layer blocks for generating the second chroma output, followed by a 3×3×Kc×Lc convolutional block. The method according to claim 1.

6. One of the one or more chroma hidden layer blocks is: 1×1×Kc×Mc CONV network, followed by a non - linear activation block, followed by a 1×1×Mc×Kc CONV network, followed by a 3×3×Kc×Kc CONV network, The method according to claim 5.

7. The method according to claim 4 or 6, wherein the 3×3 CONV network is replaced by a CP decomposition network.

8. The CP decomposition network is: a point - wise convolution network of 1×1×K×R, followed by a separable convolution network of 3×1×R×R, followed by a separable convolution network of 1×3×R×R, followed by a point - wise convolution network of 1×1×R×K, The method according to claim 7.

9. If a 1×1×M×K network is in front of the CP decomposition network, the previous 1×1×M×K network and the first 1×1×K×R CP decomposition network can be fused into a single 1×1×M×R network, the method according to claim 8.

10. If a 1×1×K×M network follows the CP decomposition network, the last 1×1×R×K CP decomposition network and the subsequent 1×1×K×M network can be fused into a single 1×1×R×M network, the method according to claim 8.

11. The method according to claim 8, wherein the separable convolution network of 3×1×R×R and the subsequent separable convolution network of 1×3×R×R are fused together to form a single separable 3×3×R×R convolution network.

12. The 3×3 convolution (CONV) network having S inputs and M outputs is replaced by a second CP decomposition network (420A), and the second CP decomposition network has: four layers of point - wise convolution of 1×1×10×R, followed by a separable convolution of 3×1×R×R, followed by a separable convolution of 1×3×R×R, followed by a point - wise convolution of 1×1×R×M, The method according to claim 2.

13. The method according to claim 2, wherein the neural network has an output 3×3×K×L convolution network after the last hidden layer.

14. The output 3×3×K×L convolutional network (420B) is decomposed into four layers including a pointwise convolution of 1×1×K×R, followed by a separable convolution of 3×1×R×R, followed by a separable convolution of 1×3×R×R, and followed by a pointwise convolution of 1×1×K×L, according to the method of claim 13.

15. The neural network has an output 3×3×Ky×Ly convolutional network after the last hidden layer, according to the method of claim 2.

16. The output loom 3×3×Ky×Ly convolution (420D) is decomposed into four layers including a pointwise convolution of 1×1×Ky×Ry, followed by a separable convolution of 3×1×Ry×Ry, followed by a separable convolution of 1×3×Ry×Ry, and followed by a pointwise convolution of 1×1×Ky×Ly, according to the method of claim 15.

17. The 3×3×Kc×Lc convolutional block (420C) is decomposed into four layers including a pointwise convolution of 1×1×Kc×Rc, followed by a separable convolution of 3×1×Rc×Rc, followed by a separable convolution of 1×3×Rc×Rc, and followed by a pointwise convolution of 1×1×Kc×Lc, according to the method of claim 5.

18. One or more components of the neural network are calculated using a fixed-point implementation, and the fixed-point implementation is: Calculating the geometric mean (GM) of all the maximum absolute value weights in the one or more hidden layer blocks (755); a) Determining the maximum absolute value weight (GlobalMax) in the "i"-th hidden layer block among the one or more hidden layer blocks; b) Setting the propagation factor (P) to 1; c) Determining the weight scaling factor (W) as W = GlobalMax / GM; d) Normalizing the "i"-th hidden layer block based on W and updating P = P*W; c) Performing forward propagation normalization of other hidden layers based on P; d) Performing backward propagation normalization of other hidden layers based on P if P > 1; e) Repeating steps a) to d) until each of the maximum absolute value weights in the one or more hidden layer blocks is equal to the geometric mean including, according to the method of claim 4.

19. The step of performing forward propagation normalization is: Scaling the bias value in layer "k" (k = i + 1... j - 1), y k = A k x k + B k ⇒ y k ' = A k x k + (B k / W) which is a step of scaling with Here, A k and B k represent the weights and bias values in the hidden layer block k, y k represents the initial output, x k represents the hidden layer input, and y k ' represents the normalized output. Layer j is determined as the layer whose maximum absolute value weight (Amax j ) is smaller than GM, in a step; the weight value in layer j as y j = A j x j + B j = (A j * M j ) * y i '+ B j = (A j * M j ) x j '+ B j a step of scaling with Here, M j = min(P, GM / Amax j ) is the step; P = P / M j and a step of setting it to; including the step of propagating P to subsequent hidden layers until P = 1 or until the last hidden layer is reached, The method according to claim 18.

20. The step of performing reverse propagation normalization is: Scaling the bias value in layer "k" (k = j + 1... i - 1) as y k = A k x k + B k ⇒ y k ' = A k x k + (B k * W) a step of scaling with Here, A k and B k represent the weights and bias values in the hidden layer block k, y k represents the initial output, x k represents the hidden layer input, and y k ' represents the normalized output. Layer j is determined as the layer whose maximum absolute value weight (Amax j ) is smaller than GM, in a step; the weight value in layer j as y j = A j x j + B j ⇒ y j ' = y j * M j = (A j * M j )x j + (B j * M j ) a step of scaling with Here, M j = min(P, GM / Amax j ) is a step; P = P / M j a step of setting it to; including the step of propagating P to subsequent hidden layers until P = 1, The method according to claim 18.

21. In step d), normalizing layer i is: When propagation normalization is performed only in the forward direction (j > i), y i = A i x i + B i ⇒ y i ' = y i / W = (A i / W)x i + B i / W; otherwise when propagation normalization is performed only in the reverse direction (j < i), y i = A i x i + B i ⇒ y i ' = (A i / W)x i + B i where A i and B i represent the weights and bias values in the hidden layer block i, and y i represents the initial output, and x i represents the hidden layer input, and y i ' represents the normalized output The method according to claim 18.

22. When reverse propagation normalization follows forward propagation normalization, layer i is y i = A i 'x i + B i '⇒y i '= A i 'x i + B i '*P re-normalized as The method according to claim 21.

23. A method for processing a video sequence encoded using one or more neural networks, the method comprising: Receiving a high-level syntax indicating that loop filtering using a neural network (NNLF) for decoding the current picture is enabled; Parsing the high-level syntax to extract a parameter indicating the recommended position of an adaptive loop filter (ALF); Decoding the current picture based on the ALF position parameter to generate an output picture, wherein the ALF position parameter performing ALF filtering after the NNLF, performing ALF filtering before the NNLF, replacing the ALF filtering with a convolutional neural network (CNN) located after the NNLF, and performing ALF filtering before an improved NNLF using input ALF classification data specifying one of a method.

24. A non-transitory computer-readable storage medium storing computer-executable instructions for executing the method according to any one of claims 1 to 23 on one or more processors.

25. An apparatus comprising a processor and configured to execute the method according to any one of claims 1 to 23.

Images