Loop filtering using neural networks

Abstract

Methods, systems, bitstream syntax, and fixed point implementations are described for loop filtering using neural networks in image and video processing. Given an input image, a hybrid luma-chroma filter is proposed in which luma and chroma components are first processed by a first neural network and then the output of the first network is processed by separate luma and chroma sub-networks. Finally, the outputs of the individual luma and chroma sub-networks are connected to generate a filtered output of the input image. Efficient computing methods using CP decomposition are also described. Methods of indicating the position of a neural network loop filter relative to other filters, such as adaptive loop filters (ALFs), are also discussed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the following applications: Indian Provisional Patent Application No. 202241038279 filed on July 4, 2022; Indian Provisional Patent Application No. 202241074543 filed on December 22, 2022; Indian Provisional Patent Application No. 202341017121 filed on March 14, 2023; and U.S. Provisional Patent Application No. 63 / 432,613 filed on December 14, 2022. Technical Field

[0003] This document relates generally to images. More specifically, embodiments of the invention relate to filtering images using neural networks. Background of the Invention

[0005] In 2020, the MPEG expert group in the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly released the first version of the Universal Video Coding standard (VVC), also known as H.266 (reference [8]). Recently, the same joint team (JVET) and still picture compression (JPEG) experts have begun to work on the development of next-generation coding standards that will provide better coding performance than existing image and video coding technologies. As part of this research, coding technologies based on artificial intelligence and deep learning are also examined. As used herein, the term "deep learning" refers to a neural network with at least three layers, and preferably more than three layers.

[0006] As appreciated by the inventors herein, improved techniques for encoding images and videos based on neural networks are described herein.

[0007] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0009] Figure 1 Depicting an example architecture for joint luma-chroma loop filtering using a neural network (NN) according to the prior art;

[0010] Figure 2Depicting an example architecture for separate luma-chroma loop filtering using a NN according to the prior art;

[0011] Figure 3 Depicting an example architecture for hybrid luma-chroma loop filtering using a NN in accordance with an embodiment of the present invention;

[0012] Figure 4A depicts an example network architecture with CP decomposition according to an embodiment of the present invention;

[0013] Figure 4B Depicting an example network architecture combining CP decomposition and fusion of 1×1 convolutional layers according to an embodiment of the present invention;

[0014] Figure 4C Describes an embodiment of the present invention Figure 4B The network architecture of , in which the input and output 3×3 convolutional layers of the neural network are additionally decomposed by CP;

[0015] Figure 4D Describes an embodiment of the present invention Figure 3 Network architecture with CP decomposition;

[0016] Figure 4E Describes an embodiment of the present invention Figure 4D The network architecture of , in which an additional CP decomposition is performed on the input and output 3×3 convolutional layers of the neural network;

[0017] Figure 4F Describes an embodiment of the present invention Figure 4B A network architecture in which the 3×1 separable convolutional layer and the 1×3 separable convolutional layer are fused into a single 3×3 separable convolutional layer;

[0018] Figure 5A Describes a network architecture for VVC loop filtering using a joint luma / chroma NNLF model according to an embodiment of the present invention;

[0019] Figure 5B Describing a network architecture for VVC loop filtering using separate luma / chroma NNLF models according to an embodiment of the present invention;

[0020] Figure 6 depicts an example of a fixed-point representation of numbers;

[0021] Fig. 7A depicts an example of local hidden layer normalization according to an embodiment of the present invention;

[0022] Figure 7Bdepicts an example process for global normalization based on forward propagation of maximum weights and normalization; and

[0023] Figure 7C Depicted is an example procedure for global normalization based on geometric mean and normalized bidirectional propagation. DETAILED DESCRIPTION

[0024] Example embodiments for using neural networks for loop filtering in image and video coding are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it is apparent that various embodiments of the present invention may be practiced without these specific details. In other cases, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring, or confusing embodiments of the present invention. Summary of the invention

[0025] Example embodiments described herein relate to image and video encoding using neural networks. In an embodiment, a processor receives an input image having a luma color component and a chroma color component, and:

[0026] applying a first neural network (NN) to both the luma color component and the chroma color component of the input image to generate a first luma output and a first chroma output;

[0027] applying a second neural network to the first luminance output to generate a second luminance output;

[0028] applying a third neural network to the first chrominance output to generate a second chrominance output; and

[0029] The second luma output and the second chroma output are concatenated to generate a color filtered output.

[0030] In a second embodiment, a processor receives a high-level syntax indicating that neural network loop filtering (NNLF) is enabled to decode a current picture. The processor then:

[0031] parsing the high level syntax to extract parameters indicating recommended positions of an adaptive loop filter (ALF); and

[0032] Decode the current picture to generate an output picture based on ALF position parameters, where the ALF position parameters specify one of the following:

[0033] ALF filtering is performed after NNLF;

[0034] Perform ALF filtering before NNLF;

[0035] Replace the ALF filter with a convolutional neural network (CNN) after the NNLF; and

[0036] ALF filtering is performed before the enhanced NNLF, which uses the ALF classification data as NNLF input.

[0037] Example encoding model using deep learning

[0038] Image and video compression methods based on deep learning are becoming increasingly popular and are an active area of ​​research. Current research on neural network (NN) based coding can be divided into two general frameworks: "hybrid" neural network based frameworks (e.g., Ref. [1]), which simply replace one or more existing encoding or decoding modules with their corresponding neural network based implementations, where each NN module is independently trained and optimized; and "end-to-end" neural networks, where training and optimization are done on the entire network (Ref. [9]). The proposed neural network filter is applicable to either architecture. The term YUV420 refers to the luminance-chrominance color space, where the chrominance is subsampled in both the horizontal and vertical dimensions, such as YCbCr 4:2:0, etc. Although the examples involve a neural network loop filter (NNLF), these techniques are applicable to various other NN based filters that remove noise artifacts or improve image quality, such as NN post-filtering, super-resolution filtering, etc.

[0039] Existing NNLF designs (references [2-4]) offer improvements of approximately 5% to 10% in coding efficiency (BDrate), but the computational cost is very high, ranging from 33 to 625 kMAC (thousands of multiply-accumulate) operations per pixel. The embodiments presented herein provide improved coding efficiency at a reduced computational cost.

[0040] Hybrid Luminance-Chroma NN Model

[0041] In the current literature, NNLF models can be divided into two main categories: joint luminance / chrominance models and separate luminance / chrominance models. These categories are based on whether the luminance (Y) signal and the chrominance (UV) signal share the same NN. For the joint model, two typical methods are used to feed the YUV420 signal into the NN. In reference [3], for example, Figure 1 As depicted, the Y signal is interleaved with the UV signal and a total of six planes are used. In addition to these six planes, the Y, U, and V boundary strength maps (BS Y , B.S. U , B.S. V ) and four additional planes of slice quantization parameters (QStep) to form 10 input channels, such as Figure 1 As shown. Figure 1 As depicted in , the architecture contains n filter blocks, including n-2 filter blocks used as hidden layers. The first layer (105) uses a 3×3 convolutional layer with an activation layer (e.g., leaky ReLU) and outputs K channels (feature maps). Each hidden filter block (110) consists of two 1×1 convolutional layers (with an activation layer (e.g., leaky ReLU) between them), followed by a 3×3×M×K convolutional layer, where the value of M is set to a value greater than K, and the notation C×C×M×K represents a C×C convolutional layer with M inputs and K outputs. By setting different values ​​for n, M, and K, models with different complexities can be created, for example, n=13, M=72, K=24, or n=14, M=216, K=72. The last layer outputs a filtered luminance-chrominance residual channel (L=6), which consists of 4 luminance planes (Y signals) and two chrominance planes (UV signals). These are added back to the NNLF input samples to obtain the final filtered output, which is used as input to the adaptive loop filter (ALF) in reference [3].

[0042] The input size of the NN filtering process is 144×144, including the current coding tree unit (CTU) and 8 neighboring samples on each side of the current CTU. The luma samples are interleaved into four 72×72 blocks before being used as input to the filtering process. The output tensor corresponds to the filtered CTU samples organized into 64×64 blocks by cropping the central region of the 72×72 filtered output (including 4 luma blocks and / or 2 chroma blocks). The output tensor of the NNLF corresponds to the filtered luma-chroma residual channels, which are scaled and added back to the NNLF input samples to obtain the final filtered output in reference [3]. During the training phase, no scaling is applied and a weighted L2 loss function is used to minimize the mean square error (MSE) between the filtered output and the original samples. During the inference phase, the scaling parameters for luma and chroma are signaled by the encoder in the slice header to minimize the mean square error (MSE) between the filtered samples and the original samples of the current frame or image.

[0043] Alternatively, as described in reference [4], the chroma (UV) samples are first upsampled to match the resolution of the Y component, and then all three components are fed to a single NN filter. At the output of the NN filter, the chroma components are downsampled back to their original (input) resolution (e.g., in 4:2:0 format).

[0044] In separate luminance / chrominance NN filters, such as Figure 2As described in (reference [2]), the Y signal and the UV (CbCr) signal use two different and separate NNs to generate the filtered output (Y'Cb'Cr'). In some embodiments, the chroma NN can also use an auxiliary luma input (Auxiliary Y).

[0045] Figure 3 An example split architecture for luma / chroma filtering according to an embodiment is depicted. The input to the network is a Y and UV joint signal. For the NN, a portion of the network is shared by both luma and chroma, and a portion of the network is split into two separate paths for the luma component and the chroma component. The corresponding luma / chroma channels may also be different. The reason behind this is that the characteristics of the luma signal and the chroma signal are different. Therefore, the chroma component may require fewer channels and / or fewer layers than the luma component without losing performance. This is why separate paths may be able to achieve similar performance while reducing complexity. In addition, because the luma / chroma signals still have some cross-component correlation, a joint path at the beginning or end of the network can help exploit this correlation.

[0046] As an example, in Figure 3 In , the architecture combines the following NN components:

[0047] Input N×N, Y×4+U+V, BS information (x3), QStep; (for example, 72×72×10)

[0048] Combined input paths:

[0049] (3×3) convolution, 3×3×10×M (10 inputs, M outputs)

[0050] Leaky ReLU

[0051] (public hidden layer)

[0052] Convolution, 1×1×M×M

[0053] Leaky ReLU

[0054] Convolution 1×1×M×K

[0055] Convolution 3×3×K×K

[0056] Luminance path, n y Hidden layers:

[0057] Convolution, 1×1×Ky×My

[0058] Leaky ReLU

[0059] Convolution 1×1×My×Ky

[0060] Convolution 3×3×Ky×Ky

[0061] Then comes the convolution, 3×3×Ky×Ly

[0062] Chroma path, n c Hidden layers:

[0063] Convolution, 1×1×Kc×Mc

[0064] Leaky ReLU

[0065] Convolution 1×1×Mc×Kc

[0066] Convolution 3×3×Kc×Kc

[0067] Then comes the convolution, 3×3×Kc×Lc

[0068] Combined output paths

[0069] Channel splicing

[0070] Output: N'×N', Y'×4+U'+V' (e.g., 64×64×6)

[0071] and Figure 1 In comparison, the following differences are also worth noting: in the first layer (305), instead of using a 3×3×10×K convolutional layer (see 105), a 3×3×10×M convolutional layer is used, and in the common hidden layer (310), instead of using a 1×1×K×M convolutional layer (see 110), a 1×1×M×M convolutional layer is used.

[0072] In an embodiment, using n in the brightness path y hidden layers and n in the chromaticity path c With 1 hidden layer, the complexity of the proposed architecture (in kMAC / pixel) can be calculated as follows:

[0073] Total MAC in the brightness path = n y *(2*K y *M y +9*K y *K y )+9*K y *L y

[0074] Total MACs in the chroma path = n c *(2*K c *M c +9*K c *K c )+9*K c *L c

[0075] Total MAC in the public path = 9*10*M+M*M+K*M+9*K*K

[0076] For example:

[0077] If in the public path: N = 72, K = 24, M = 72, then MAC = 18,576

[0078] If in the brightness path: n y =10, K y =16,M y =48,L y =4, then MAC=38,976

[0079] If in the chroma path: n c =10, K c =8,M c =24,L c =2, then MAC=9,744

[0080] Total MAC = 67,296 processing 4 luma (and 2 chroma) samples, and ignoring 72x72 boundary pixels in the output, resulting in 64x64.

[0081] Therefore, MAC / pixel = (67,296*72*72) / (64*64*4) = 21.292 kMAC / pixel. Compared to reference [3] (which reports a complexity of about 33.6 kMAC / pixel), the proposed architecture requires approximately 0.633 of the computational effort.

[0082] Tensor Decomposition Method

[0083] Applying tensor decomposition is one of many methods for reducing the computational complexity of NNs. There are several tensor decomposition methods. For example, the depthwise separable convolution (DSC) method has been attempted in several JVET papers such as reference [5]. In an embodiment, CP decomposition is used to reduce NN complexity (reference [6]). In CP decomposition (CP is derived from the CANDECOMP / PARAFAC model), a 4D convolution kernel tensor is decomposed into a sequence of four convolutional layers with small kernels. The first convolution layer is a point-by-point convolution, the second and third layers are spatial convolutions in the X and Y directions, and the fourth convolution is also a point-by-point convolution in the channel dimension.

[0084] The regular convolution with output channel t can be written as

[0085]

[0086] where U is the input tensor with S channels, K is a kernel of size d×d×S for each output channel, and

[0087] V is the output tensor.

[0088] For channel t, the CP rank R of the above convolution is approximately given by:

[0089]

[0090] Here, the kernel K is approximated as

[0091]

[0092] Among them, K x , K x , K y , K s , K t , K t are d×R, d×R, S×R, and T×R tensors along different dimensions. 2 In contrast, the complexity of CP decomposition (expressed in MAC / pixel) is R(S+2d+T). If the recommended rank is chosen If decomposed, the improvement factor will be d 2 For training, CP decomposition can reuse the trained model and fine-tune it using regular convolutions.

[0093] As an example, Figure 4A Depicted in Figure 1 An example of applying CP decomposition (420) to the architecture depicted in Figure 3 Modifications are made in the common layers of , namely: in the first layer, instead of using a 3×3×10×K convolution layer (see 105), a 3×3×10×M convolution layer is used, and in the first hidden layer, instead of using a 1×1×K×M convolution layer (see 110), a 1×1×M×M convolution layer is used.

[0094] like Figure 4A As depicted in (details of 420), the 3×3 convolutions of each hidden layer are decomposed into four layers with rank R (see 420):

[0095] Layer 1: 1×1×K×R point-wise convolution

[0096] Layer 2: 3×1×R×R separable convolutions

[0097] Layer 3: 1×3×R×R separable convolution

[0098] Layer 4: 1×1×R×K point-wise convolution

[0099] Compared with reference [3], when R = 24, applying CP decomposition to all hidden layers reduces the complexity to about 1 / 1.6. When using R = 12, the complexity is reduced to about 1 / 1.8.

[0100] like Figure 4A and Figure 4B As depicted in FIG, in the CP decomposition (420), the first decomposition layer (415) (1×1×K×R) can be fused with the 1×1×M×K (405) located before the 4 layers of hidden units to produce an effective fused single layer (430) (1×1×M×R). Similarly, the last CP decomposition layer (425) (1×1×R×K) can be fused with the first layer (410) (1×1×K×M) of the next hidden unit to achieve the same function through a single fused (1×1×R×M) layer (440). Figure 4B This fusion of adjacent 1×1 point-wise convolutions is illustrated in . This fusion is possible because there are no non-linear activations between the layers to be fused, further reducing the complexity. Due to differences in the order of floating point operations, Figure 4B The fused network in may result in small differences in the inference output, but is similar to Figure 4A Compared with the CP decomposition NNLF filter without fusion, the difference in coding performance is negligible.

[0101] Compared with reference [3], when R=24, the complexity is reduced to about 1 / 2.1 by applying CP decomposition and fusion to all hidden layers. When R=12 is used, the complexity is reduced to about 1 / 3.3.

[0102] Figure 4C Depicted Figure 4B , wherein an additional CP decomposition is performed on the input 3×3 convolution layer (420A) and the output 3×3 convolution layer (420B). The input 3×3×10×M convolution layer (420A) is decomposed into the following 4 layers: 1×1×10×R point-wise convolution, 3×1×R×R separable convolution, 1×3×R×R separable convolution, and 1×1×R×M point-wise convolution. Similarly, the output 3×3×K×L convolution (420B) is decomposed into the following 4 layers: 1×1×K×R point-wise convolution, 3×1×R×R separable convolution, 1×3×R×R separable convolution, and 1×1×K×L point-wise convolution. In the case of R=24, by performing additional decomposition on the input and output layers, the overall complexity is now reduced to about 1 / 2.3.

[0103] Figure 4D Describes the method for applying CP decomposition to Figure 3Example embodiment of a segmentation architecture. As discussed above, the CP decomposition layer (420) can also be fused with a previous or subsequent 1×1 convolution layer (if applicable). CP decomposition of rank R=24 applied to all hidden layers achieves a complexity reduction of 1 / 2.5. Applying CP decomposition of rank 24 to all hidden layers and fusing adjacent 1×1 point-by-point convolutions achieves a complexity reduction of 1 / 3.

[0104] Figure 4E Depicted Figure 4D The network architecture of FIG. 4 is shown in FIG. 4 , wherein the input 3×3 convolution layer (420A) and the luminance and chrominance output 3×3 convolution layers (420D, 420C) are additionally decomposed by CP. The input 3×3×10×M convolution layer (420A) is decomposed into the following 4 layers: 1×1×10×R point-wise convolution, 3×1×R×R separable convolution, 1×3×R×R separable convolution, and 1×1×R×M point-wise convolution. The output luminance 3×3×Ky×Ly convolution (420D) is decomposed into the following 4 layers: 1×1×Ky×Ry point-wise convolution, 3×1×Ry×Ry separable convolution, 1×3×Ry×Ry separable convolution, and 1×1×Ky×Ly point-wise convolution. Similarly, the output chroma 3×3×Kc×Lc convolution (420C) is decomposed into the following 4 layers: 1×1×Kc×Rc point-wise convolution, 3×1×Rc×Rc separable convolution, 1×3×Rc×Rc separable convolution and 1×1×Kc×Lc point-wise convolution. In the case of R=24, Ry=16 and Rc=8, the overall complexity is reduced to about 1 / 3.4 by additional decomposition of the input and output layers.

[0105] like Figure 4F As depicted in Figure 4B The decomposed separable convolution layers 3×1 (417A) and 1×3 (417B) depicted in FIG. 4 can be fused into a single 3×3 separable convolution (445). This fusion helps reduce the number of layers in the network, which can help reduce the latency and memory access overhead of the network, and can utilize hardware / software implementations that are optimized for 3×3 convolution kernel size but not optimized for 3×1 or 1×3 row-by-row or column-by-column convolution operations. This fusion is possible because there are no non-linear activations between the fused layers and the fused block is mathematically equivalent to the fused layer. The additional 'mac' (multiply-accumulate) operations caused by the fusion (9×R×R in the fused layer compared to 6×R×R before fusion) have a negligible impact on the overall complexity of the filtering. Figure 4F The resulting 'mac complexity' is less than Figure 4B 2%, while an implementation-specific speedup of 30% was observed.

[0106] NNLF and ALF pipeline issues

[0107] When NNLF is used in the VVC framework (reference [8]), the location of NNLF is very important. VVC contains several in-loop filters, such as inverse luma mapping followed by chroma scaling (LMCS), deblocking filter (DF), sample adaptive offset (SAO), adaptive loop filter (ALF) and cross-component ALF (CCALF), which are applied to improve the quality of the decoded signal. When NNLF is proposed, DF / SAO can be incorporated into NNLF. So far, all JVET contributions cannot eliminate ALF and CCALF because they bring additional gain. In addition, since ALF and CCALF need to calculate coefficients based on the distortion between the decoded / filtered signal and the original signal, it is found that the highest gain can be achieved by placing ALF and CCALF after NNLF; however, the placement of ALF and CCALF filters may cause pipeline problems because in practical applications, NNLF is most likely implemented by a graphics processing unit (GPU), while ALF, CCALF and other decoding modules are implemented by a CPU. In order to solve this pipeline problem, the following embodiment is described.

[0108] First embodiment: Replace ALF and CCALF with convolutional NN (CNN)

[0109] Tests show that the main gains from ALF and CCALF are for chroma components. Conceptually, CNN is essentially ALF and CCALF. In an embodiment, a separate CNN can be added after NNLF. More specifically, a separate CNN may include only a layer that performs a convolution or dot product of a convolution kernel with an input matrix of the layer. Such a CNN can be pre-trained or trained online (so-called CNN model update). For the latter method, the coefficients need to be sent with the bitstream. For intra-pictures or inter-pictures referenced by other pictures, the updated coefficients can be notified by signal.

[0110] Second embodiment: Enhanced NNLF

[0111] Another approach is to use NNLF filters as the last stage of all loop filters, but add input ALF classification data. This architecture will be denoted as 'Enhanced NNLF'. The advantage of this approach is that it does not affect the hardware pipeline of the conventional loop filters in VVC (e.g., deblocking, SAO, and ALF). This approach uses 4×4 block classification metrics based on horizontal, vertical, diagonal gradients, and local activity as additional feature inputs to the NNLF filters, which are used in the current VVC ALF filters. In addition to this, separate NNLF models can be used to process luma and chroma.

[0112] Figure 5AEmbodiments for VVC loop filtering using a joint (or hybrid) NNLF network are described. Figure 5A As depicted in , the reconstructed luma samples (RecY) are processed sequentially through inverse luma mapping (inverse LMCS), deblocking, SAO, and ALF filtering to generate luma ALF classification data (502) and luma ALF samples (504). The output of the luma SAO is also processed by the Cb CCALF filter and the Cr CCALF filter.

[0113] The reconstructed chroma samples (RecCb, RecCr) are processed sequentially through deblocking, SAO and ALF, and their outputs are added to the corresponding outputs of the Cb / Cr CCALF filter to generate Cb ALF samples (506) and Cr ALF samples (508). Finally, the ALF classification data (502), luma ALF samples (504), chroma AL samples (506, 508) are combined together with the BS information and QStep data as the input of the NNLF to generate the filtered output Y'Cb'Cr'.

[0114] Figure 5B Describes an alternative embodiment for VVC loop filtering using separate luma NNLF networks and chroma NNLF networks. The previous part of the design is similar to Figure 5A ; however, the generated data (e.g., 502, 504, 506, and 508) is fed to two separate networks. Figure 5B As depicted in , the luma NNLF filter does not use any chroma samples; however, the chroma NNLF filters may use the filtered luma ALF samples (504) as part of their input.

[0115] Syntax Examples

[0116] The proposed tool may be communicated from the encoder to the decoder using a high level syntax (HLS), which may be part of a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), slice header (SH), or as part of supplementary metadata such as supplementary enhancement information (SEI) data. An example syntax is depicted in Table 1. Alternatively, such signaling may not be required if the specific architecture is predetermined and known by both the encoder and decoder.

[0117] Table 1. Example of high-level syntax for NNLF adaptation

[0118]

[0119] nnlf_adaptation_enabled_flag equal to 1 specifies that NNLF adaptation is enabled for the decoded picture. nnlf_adaptation_enabled_flag equal to 0 specifies that NNLF adaptation is not enabled for the decoded picture.

[0120] hybrid_luma_chroma_model_idc identifies the NNLF model. hybrid_luma_chroma_model_idc equal to 0 specifies that the NNLF does not apply a hybrid luminance-chrominance model. hybrid_luma_chroma_model_idc equal to 1 specifies that the NNLF may apply a hybrid luminance-chrominance model. Figure 3 The hybrid luminance-chroma model shown. Other values ​​of hybrid_luma_chroma_model_idc are reserved for future use.

[0121] tensor_decomposition_enabled_flag equal to 1 specifies that tensor decomposition is used to reduce NNLF complexity. tensor_decomposition_enabled_flag equal to 0 specifies that tensor decomposition is not used to reduce NNLF complexity.

[0122] tensor_decomposition_idc identifies the tensor decomposition method specified in Table 2. Other values ​​of tensor_decomposition_idc are reserved for future use.

[0123] Table 2. tensor_decomposition_idc values

[0124] value illustrate 0 CP decomposition 1 Depthwise Separable Convolution 2 Optimal Separable Convolution 3 Low rank decomposition

[0125] cp_decomp_rank_minus1 plus 1 specifies the rank of CP decomposition.

[0126] ALF_placement_idc identifies the ALF placement as specified in Table 3.

[0127] Table 3. ALF_placement_idc values

[0128]

[0129] Fixed-point implementation

[0130] Embodiments of the present disclosure propose different methods to quantize the floating-point convolution weights and bias layers of a neural network to achieve an efficient fixed-point neural network without affecting its performance in terms of output accuracy.

[0131] The main goals of a fixed-point integer implementation of any neural network are to:

[0132] 1. Bit-accurate inference: Achieve exactly the same output for a given input on any hardware or software platform.

[0133] 2. Low complexity: Software and hardware friendly implementation using integer arithmetic operations, which aims to reduce complexity, improve throughput and energy efficiency.

[0134] 3. Accuracy: Maintain the highest possible accuracy and achieve the smallest possible deviation compared to floating-point inference.

[0135] Although all methods in this disclosure are applicable to any convolutional feedforward neural network, the descriptions and embodiments are implemented using the low-complexity neural network-based loop filter (NNLF) described previously.

[0136] Since loop filtering is usually a canonical process in any video decoder, it is highly desirable for the decoder to have bit-accurate output on any hardware / software platform. Since bit-accurate output is not achievable in floating-point inference, a fixed-point implementation is required. For example, the NNLF floating-point model is initially trained in PyTorch or TensorFlow and is subsequently converted to fixed-point in the process.

[0137] All methods proposed in this invention are aimed at implementing integer implementations of NNLF. Without limitation and as an example, integer implementation and verification are performed using SADL (Small Self-Organizing Deep Learning Library), which is a lightweight library for performing neural network inference in pure C++. In addition, in general, 16-bit fixed-point integer implementations (int16) are more ideal than 32-bit integer implementations (int32) to achieve faster implementations and lower storage costs. Without limitation and as an example, for the full-frame Intra-Main 10 of the JVET "Class D" test image set, the performance of the fixed-point implementation will be measured based on its performance compared to its floating-point implementation, as shown in Table 4. For example, in a floating-point implementation, the trained NNLF filter can improve the Y-PSNR by 5.04% compared to a VVC encoder without a NNLF filter.

[0138] Table 4. Example NNLF floating point performance

[0139]

[0140] Figure 6An example of a fixed-point representation of a number is depicted. The K bits before the virtual radix point represent the integer portion, including the sign bit, and the F bits after the radix point represent the fractional portion. The Q format or Q factor of the fixed-point representation is determined by the number of F bits representing the fractional portion, so, for example, the term Q14 represents a fixed-point implementation with F=14 bits. Thus, for Int16, Q11 indicates that there are 5 bits for the sign and integer portion, and 11 bits for the fractional portion, for a total of 16 bits.

[0141] A high-accuracy and efficient fixed-point implementation of NNLF involves determining the optimal Q-format for the weights and biases of all convolutional layers as well as the inputs and intermediate outputs of each layer and the final output of the NNLF filters.

[0142] All Q-format related operations in fixed-point convolution are explained as follows:

[0143] 1. Normalize the input x(0,1) and multiply it by the optimal Q factor based on the required accuracy (such as Q in ).

[0144] 2. Each convolution weight (A) and bias (B) are also determined by their own Q factor Q A and Q B express.

[0145] 3. It is expected that the resulting convolution output y = Ax + B also uses Q in Therefore, the multiplication and accumulation results of y' = Ax are calculated according to the Q factor Q of the convolutional layer. A Shift right to return the output y' to the input Q format Q in .

[0146] 4. In the bias addition layer y = y' + B, before adding, the difference (Q in -Q B ) scales the input y' up or down to match the biased Q format Q B It is desirable that the Q format of the bias layer is Q B and input Q in are identical, so there is no loss of precision or data overflow due to the shift introduced in the addition.

[0147] All remaining sections show implementations for int32 and int16 fixed-point implementations and the corresponding PSNR results.

[0148] Table 5 depicts some experimental results using various fixed-point implementations and their comparison with the floating-point results of Table 4. The term (w) refers to the weight, and the term (b) refers to the bias.

[0149] For Int32, Q14, the results with this configuration closely match the reference floating point results. When using Q11, as expected, a performance loss is detected. If dynamic Q is applied to the weights, but the Q11 of the bias is fixed, the performance improves; however, if a more ideal 16-point implementation is applied, a significant performance loss is observed due to overflow of the variables during the convolution operation.

[0150] The convolution weights and biases of some layers have very high magnitudes. These values ​​are mainly seen in the 1×1 convolutional layers. These values ​​are very high in the floating point domain. Therefore, the magnitude of the output of the layers tends to increase even after right shifting by the Q factor of the layer. This means that there are a few convolutional layers where the output energy is higher than the input energy. This increase in magnitude will lead to overflows in subsequent convolutional layers.

[0151] Table 5. Example NNLF fixed-point performance

[0152]

[0153] Int16 implementation based on dynamic weight renormalization

[0154] In principle, the output of the NNLF filter after all convolutional layers will have similar (or lower) energy than the input. If a few layers in the model have large-magnitude convolutional weights that cause amplification, then several other layers in the subsequent pipeline will have much smaller weights, reducing the dynamic range to that of the input. Since high-magnitude convolutional layers can cause overflow, it is desirable to normalize the convolutional weights of such a layer (e.g., i) by a scaling factor W (where W>1) to prevent overflow, as shown in the following equation:

[0155]

[0156] Since the output y i′ Now normalized by W, we need to rescale this factor in subsequent (or previous) layers (e.g., j) where the convolution weight magnitude is small enough, as shown in the following equation to ensure that y j =y j′ :

[0157]

[0158] Note that for j=i+1, the bias term is not rescaled as per the derivation shown below:

[0159] y j =A j x j +B j =A j y i +B j =(Aj *W)*y i′ +B j =(A j *W)x j′ +B j (6)

[0160] In summary, layers with larger magnitudes are divided by the scaling factor, while layers with smaller magnitudes are multiplied by these scaling factors. These normalizations can be applied in such a way that the NNLF floating-point output is mathematically the same before and after normalization.

[0161] Note: In the above explanation, if the scaling factor W propagates to a subsequent layer (j > i + 1) that is not immediately after the current layer (i), then all intermediate bias layers between i and j also need to be normalized by W. This is specified in the following equation (7):

[0162]

[0163] Note that in equation (7), the convolutional weights are not modified and only the bias terms are normalized.

[0164] The factor propagated to a subsequent layer (j > i + 1) for rescaling is also referred to as the propagation factor P in methods 2 and 3 explained below.

[0165] Equations (4) to (7) are only applicable when normalization is performed in the forward direction (j > i). If normalization is performed in the reverse direction (j < i), then the weights and biases of the current layer (i) and the previous layer are modified as shown in the following equations (8), (9), and (10):

[0166]

[0167] Note: Further details on when to apply equations (8) - (10) will be described later as part of the proposed method 3.

[0168] Based on this weight normalization principle, the following three different methods were tested for 16-bit fixed-point inference:

[0169] 1. Local hidden layer normalization.

[0170] 2. Global normalization based on forward propagation with maximum weight sum normalization.

[0171] 3. Global normalization based on two-way propagation with geometric mean normalization.

[0172] Method 1: Local hidden layer normalization

[0173] This method is described in Fig. 7A and includes the following steps:

[0174] 1. Divide the second 1×1 convolutional layer (e.g., 705) of each hidden layer by a factor W based on the highest magnitude weight of that layer, so that the resulting magnitude is constrained to be less than a fixed threshold (e.g., threshold=4)

[0175] 2. By factoring the weights of the next 3×1 and 1×3 (710, 715) convolutional layers of the same hidden layer Scaling is used to achieve this division.

[0176] 3. First, perform floating point inference verification on the renormalized network, and then perform int16 fixed point inference verification using dynamic Q factor of convolution weights and fixed Q11 of bias.

[0177] The results of this approach are shown in Table 6. The results of Approach 1 show that the int16 encoding performance has improved considerably, but is still significantly lower than the performance of int32 with fixed Q11 with dynamic Q and bias (see Table 5). The main conclusions and shortcomings of this approach are:

[0178] 1. Local normalization may still cause overflow.

[0179] 2. If the scaling factor W is very large due to abnormally high value weights, the normalization power is insufficient.

[0180] Method 2: Global normalization based on maximum weight and forward propagation.

[0181] This method is Figure 7B , and includes the following steps:

[0182] 1. Starting from the first hidden layer, evaluate the maximum magnitude weight of each convolutional layer. Set the initial propagation factor P to 1.

[0183] 2. Determine the division factor (Di1, Di2) of each 1×1 convolutional layer (702, 705) based on the maximum magnitude weight in that layer. The division factor is applied so that the magnitude of all 1×1 convolutional weights is constrained to be less than 1.

[0184] 3. Determine the multiplication factors (Mi1, Mi2) for subsequent 3×1 and 1×3 convolutional layers (710, 715) based on Di1, Di2, and any earlier propagation factors P>1. Constrain the maximum allowed target weights for the 3×1 and 1×3 layers to be less than a fixed value (e.g., 5).

[0185] 4. If (Mi1*Mi2) cannot fully accommodate (P*Di1*Di2), update the propagation factor P to (Di1*Di2*P) / (Mi1*Mi2) and propagate forward, otherwise if it is fully accommodated, update P to 1.

[0186] 5. Divide all future bias layers by the propagation factor (P) until they are fully computed during the process.

[0187] 6. Normalize in the forward direction until all 1×1 convolutional layers in the hidden layer are normalized.

[0188] 7. Multiply the propagation factor P remaining after the last 3×1 and 1×3 layers by the last 1×1 layer in the network, since this propagation factor cannot be propagated further.

[0189] 8. First, perform floating point inference verification on the renormalized network, and then perform int16 fixed point inference verification using dynamic Q factor of convolution weights and fixed Q11 of bias.

[0190] The results of this approach are also depicted in Table 6. Approach 2 shows considerable improvement in int16 encoding performance compared to Approach 1, but its performance is still worse than that of int32 with dynamic Q for weights and Q11 for biases.

[0191] The main conclusions and shortcomings of this method are:

[0192] 1. Arbitrary constraints on the maximum allowed amplitudes of 1×1, 1×3, and 3×1 convolutions.

[0193] 2. One-way forward propagation may not be sufficient to fully absorb large propagation factors.

[0194] Method 3: Global Normalization Based on Geometric Mean and Bidirectional Propagation

[0195] This method aims to find the geometric mean (GM) of the maximum magnitude weights of each layer and iteratively normalize the network weights so that all maximum magnitude weights are exactly scaled to the GM. Weight propagation is done in both forward and reverse directions. Given m numbers, a1, a2, ...a m , their geometric mean is defined as

[0196]

[0197] This method is Figure 7C , and includes the following steps:

[0198] 1. Calculate all maximum magnitude weights Amax from each layer i =(Max(abs(A i ))Geometric mean (GM) (755)

[0199] 2. Set the initial propagation factor P to P = 1, and GlobalMax = Max(Amaxi )(780).

[0200] 3. Determine the hidden layer 'i' with the highest magnitude weight and normalize its weights by a scaling factor W such that W = GlobalMax / GM(782). Assume forward propagation is subsequently performed (784), and adjust its bias value according to equation (4) (782). Update the propagation factor P to P = P * W.

[0201] 4. Forward propagation normalization (784): Propagate the scaling factor P to the layers in the forward direction (j > i) whose maximum weight is less than GM. Multiply the weights of layer 'j' with maximum weight Amax j less than GM by a multiplication factor M j = min(P, GM / Amax j ) to magnify such that its maximum weight after scaling is less than or equal to GM. For layer j, perform the scaling operation according to equation (6), and for layer k (i < k < j), perform the scaling operation according to equation (7). Update the propagation factor P to P = P / M j . Continue forward propagation normalization until P becomes 1 or reaches the last layer. Denote the value of P at the end of forward propagation normalization as Pb (the backward propagation factor). If Pb = P > 1, then backward propagation normalization is required (788). However, before this step, the bias value of hidden layer i needs to be readjusted as in equation (12) (786), where A i′ and B i′ represent the adjusted values after step 3 (782) using equation (4).

[0202]

[0203] 5. Backward propagation normalization (788): If the propagation factor P has not been fully computed in the forward direction (j < i), then it is propagated in the backward direction. Multiply the weights of layer 'j' with maximum weight Amax j less than GM by a multiplication factor M j = min(P, GM / Amax j ) to magnify such that its maximum weight after scaling is less than or equal to GM. For layer j, perform the scaling operation according to equation (9), and for layer k (j < k < i), perform the scaling operation according to equation (10). Update the propagation factor P to P / M j . Continue backward propagation normalization until P becomes 1.

[0204] 6. The maximum magnitude of weights on all layers (i.e., GlobalMax) changes after each iteration because GlobalMax is normalized to GM in each iteration. Hopefully, after the whole process, the maximum weight of each layer (i.e., Amax) i ) will all be equal to GM. Therefore, steps 3 to 5 (780 to 788) are repeated until all maximum magnitude weights of each layer are reduced to GM (790). For example, in the first iteration, GlobalMax may be in the i-th layer, but after the first iteration, the maximum weight of the i-th layer is GM(Amax i Some other layers that are smaller than GM will be scaled up to GM accordingly), so for the second iteration, GlobalMax will come from a different layer. This process is repeated until the maximum weight among all the layers' weights is equal to GM.

[0205] 7. First, perform floating point inference verification on the renormalized network, and then perform int16 fixed point inference verification using dynamic Q factor of convolution weights and fixed Q11 of bias.

[0206] In another embodiment, in step (782), it may be decided to start with back propagation (788). Then, for hidden layer i, its bias value needs to be adjusted based on equation (8), and steps (784) and (788) need to be swapped. In addition, the value of P after back propagation normalization is expressed as Pf (forward propagation factor). If Pf = P>1, and before forward propagation normalization begins, the bias value of hidden layer i needs to be re-adjusted as in equation (13), where A i′ and B i represents the adjusted value after using step 3 (782) of equation (8).

[0207]

[0208] The results of this approach are also shown in Table 6. Method 3 shows additional improvement in int16 encoding performance. The results are only slightly lower than the performance of int32 with dynamic Q for weights and fixed Q11 for bias values ​​(see Table 5).

[0209] Table 6. Example Int16 performance and dynamic weight normalization

[0210]

[0211] To help better understand methods 1 to 3 and without limitation, example values ​​of weights and bias values ​​in certain hidden layers before and after the proposed normalization process are provided in the Appendix.

[0212] References

[0213] Each of the references listed herein is incorporated by reference in its entirety. The term JVET refers to the Joint Video Experts Group of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC 29.

[0214] [1] Dong Liu, Yue Li, Jianping Lin, Houqiang Li, Feng Wu, “Deep learning-based video coding: A review and a case study,” available at https: / / arxiv.org / abs / 1904.12462.

[0215] [2] Y. Li, K. Zhang, L. Zhang, H. Wang, J. Chen, K. Reuze, A.M. Kotra, and M. Karczewicz, “EE1-1.6: Combined Test of EE1-1.2and EE1-1.4,” JVET-X0066, conference call, October 2021.

[0216] [3] H. Wang, J. Chen, K. Reuze, A. M. Kotra, and M. Karczewicz, “EE1-1.4: Tests on Neural Network-based In-Loop Filter with constrained computational complexity,” JVET-X0140, conference call, October 2021.

[0217] [4] L. Wang, X.,Xu, and S. Liu, “EE1-1.1: neural network based in-loop filter with constrained storage and low complexity,” JVET-Y0078, teleconference, January 2022.

[0218] [5] L. Wang, S. Lin, X. Xu, S. Liu, and X. Li, “E1-1.5: neural network based in-loop filter using depthwise separable convolution and regular convolution,” JVET-X0053, conference call, October 2021.

[0219] [6] V. Lebedev et al., “Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition,” ICLR 2015.

[0220] [7] M. Jaderberg et al., “Speeding up Convolutional Neural Networks with Low Rank Expansions,” in Proceedings of the British Machine Vision Conference (BMVC), 2014.

[0221] [8]Versatile Video Coding, Rec. ITU-T H.266, August 2020, ITU.

[0222] [9] A. Mohananchettiar et al., “Multi-distribution entropy modeling of latent features in image and video coding using neural networks,” PCT application serial number PCT / US2022 / 021730, filed on March 24, 2022.

[0223] Example Computer System Implementation

[0224] Embodiments of the invention may be implemented using a computer system, a system configured with electronic circuit systems and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may perform, control, or execute instructions related to loop filtering using neural networks in image and video encoding, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to loop filtering using neural networks in image and video encoding as described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0225] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement methods related to loop filtering using a neural network in image and video encoding as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention can also be provided in the form of a program product. The program product may include any non-transient tangible medium that carries a set of computer-readable signals including instructions, which, when executed by a data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention may be in any of a variety of non-transient and tangible forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0226] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, references to these components (including references to "means") should be interpreted as including any components that perform the functions of the described components as equivalents (e.g., functionally equivalent) to the components, including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention, unless otherwise specified.

[0227] Equivalents, extensions, alternatives and miscellaneous

[0228] Thus described are example embodiments related to loop filtering using neural networks in image and video coding. In the foregoing description, embodiments of the invention have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the sole and exclusive indication of the invention and the applicant's inventive intent is the set of claims issued in specific form according to this application, wherein such claim issuance includes any subsequent corrections. Any definitions expressly set forth herein for terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes not expressly cited in a claim should not limit the scope of such claim in any way. Therefore, the specification and drawings should be viewed in an illustrative rather than a restrictive sense.

[0229] appendix

[0230] This appendix provides example numerical results during the normalization process for each of Methods 1, 2, and 3.

[0231] Method 1

[0232] Text description: Normalization by W is shown in italics, Zoom is shown in bold

[0233] In all steps, W=8.

[0234] In step 1, hidden layer I is processed, and its coefficients remain the same for all subsequent steps. In step 2, hidden layer I+1 is processed, and its coefficients remain the same for all subsequent steps.

[0235] In step 3, hidden layer I+2 is processed and its coefficients remain the same for all subsequent steps.

[0236] Appendix Table 1. Instance normalization of hidden layers using method 1

[0237]

[0238] Method 2

[0239] Text description: Normalized by Di1 and Di2 are shown in italics, and scaled by Mi1 and Mi2 are shown in bold.

[0240] Hidden layer i (i=1, 2, ... n) is processed in step i and its coefficients remain the same for all subsequent steps.

[0241] Appendix Table 2. Instance normalization of hidden layers using method 2

[0242]

[0243] Method 3

[0244] GM = 1.8981781. Steps 1-3 are example forward propagation normalization steps. Appendix Table 3. Example normalization of hidden layers using method 1

[0245]

Claims

1. A method for performing image filtering using a neural network, the method comprising: Access an input image having a luma color component and a chroma color component; applying a first neural network (NN) to both the luma color component and the chroma color component of the input image to generate a first luma output and a first chroma output; applying a second neural network to the first luminance output to generate a second luminance output; applying a third neural network to the first chrominance output to generate a second chrominance output; as well as The second luma output and the second chroma output are concatenated to generate a color filtered output.

2. The method of claim 1, wherein: The first neural network comprises: a 3×3 convolutional (CONV) network with S inputs and M outputs (3×3×S×M), This is followed by the non-linear activation (NLA) block, followed by a 1×1×M×M CONV network and a second NLA block, and Then there are 1×1×M×K and 3×3×K×K convolutional networks, which are used to generate the first luminance output with Ky signals and the first chrominance output with Kc signals, where K=Ky+Kc.

3. The method of claim 1, wherein: The second neural network comprises: Ky luminance signal input, and One or more luma hidden layer blocks followed by a 3×3×Ky×Ly convolution block for generating the second luma output.

4. The method of claim 3, wherein one of the one or more brightness hidden layer blocks comprises: 1×1×Ky×My CONV Network, Then comes the nonlinear activation block, Then there is a 1×1×Ky×Ky CONV network, This is followed by a 3×3×Ky×Ky CONV network.

5. The method of claim 1, wherein: The third neural network comprises: An input of Kc chrominance signals; and One or more chroma hidden layer blocks, followed by a 3×3×Kc×Lc convolution block, wherein the 3×3×Kc×Lc convolution block is used to generate the second chroma output.

6. The method of claim 5, wherein: One of the one or more chroma hidden layer blocks comprises: 1×1×Kc×Mc CONV network, Then comes the nonlinear activation block, Then there is a 1×1×Mc×Kc CONV network, This is followed by a 3×3×Kc×Kc CONV network.

7. The method according to claim 4 or 6, wherein: The 3×3CONV network is replaced by a CP decomposition network.

8. The method of claim 7, wherein: The CP decomposition network comprises: 1×1×K×R point-wise convolutional network, followed by 3×1×R×R separable convolutional network, followed by 1×3×R×R separable convolutional network, followed by 1×1×R×K point-wise convolutional network.

9. The method of claim 8, wherein: If the CP decomposition network is preceded by a 1×1×M×K network, the preceding 1×1×M×K network and the first 1×1×K×R CP decomposition network can be fused into a single 1×1×M×R network.

10. The method of claim 8, wherein: If the CP decomposition network is followed by a 1×1×K×M network, the last 1×1×R×K CP decomposition network and the following 1×1×K×M network can be fused into a single 1×1×R×M network.

11. The method of claim 8, wherein: The 3×1×R×R separable convolutional network is fused with the subsequent 1×3×R×R separable convolutional network to form a single separable 3×3×R×R convolutional network.

12. The method of claim 2, wherein: The 3×3 convolution (CONV) network with S inputs and M outputs is replaced by a second CP decomposition network (420A), comprising: The following 4 layers: 1×1×10×R point-wise convolution; Then there is a 3×1×R×R separable convolution; followed by a 1×3×R×R separable convolution; and This is followed by a 1×1×R×M point-wise convolution.

13. The method of claim 2, wherein: The neural network includes an output 3×3×K×L convolutional network after the last hidden layer.

14. The method of claim 13, wherein: The output 3×3×K×L convolutional network (420B) is decomposed into 4 layers, including: 1×1×K×R point-wise convolution, followed by 3×1×R×R separable convolution, followed by 1×3×R×R separable convolution, and then 1×1×K×L point-wise convolution.

15. The method of claim 2, wherein: The neural network includes an output 3×3×Ky×Ly convolutional network after the last hidden layer.

16. The method of claim 15, wherein: The output brightness 3×3×Ky×Ly convolution (420D) is decomposed into the following 4 layers: 1×1×Ky×Ry point-wise convolution, followed by 3×1×Ry×Ry separable convolution, followed by 1×3×Ry×Ry separable convolution, and then 1×1×Ky×Ly point-wise convolution.

17. The method of claim 5, wherein: The 3×3×Kc×Lc convolution block (420C) is decomposed into the following 4 layers: 1×1×Kc×Rc point-wise convolution, followed by 3×1×Rc×Rc separable convolution, followed by 1×3×Rc×Rc separable convolution, and then 1×1×Kc×Lc point-wise convolution.

18. The method of claim 4, wherein: One or more components of the neural network are computed using a fixed point implementation, wherein the fixed point implementation comprises: Calculating (755) a geometric mean (GM) of all maximum absolute value weights in the one or more hidden layer blocks; a) determining the maximum absolute value weight (GlobalMax) in an "i" hidden layer block among the one or more hidden layer blocks; b) Set the propagation factor (P) to 1; c) determining the weight scaling factor (W) as W = GlobalMax / GM; d) normalizing the "i" hidden layer block based on W, and updating P = P * W; c) Perform forward propagation normalization on other hidden layers based on P; d) if P>1, then perform back-propagation normalization on other hidden layers based on P; and e) Repeating steps a) to d) until each of the maximum absolute value weights in the one or more hidden layer blocks is equal to the geometric mean.

19. The method of claim 18, wherein: The steps to perform forward propagation normalization include the following calculations: Scale the bias values ​​in layer "k" (k = i+1...j-1) to Among them, A k and B k represents the weight and bias value in hidden layer block k, y k represents the initial output, x k represents the hidden layer input, and y k′ represents the normalized output, and Among them, layer j is determined as its maximum absolute value weight (Amax j ) is smaller than the layer of GM; Scale the weight values ​​in layer j to y j =A j x j +B j =(A j *M j )*y i′ +B j =(A j *M j )x j′ +B j , in, Set P = P / M j ;as well as Propagate P to subsequent hidden layers until P = 1 or until the last hidden layer is reached.

20. The method of claim 18, wherein: The steps to perform backpropagation normalization include performing the following calculations: Scale the bias values ​​in layer "k" (k=j+1...i-1) to Among them, A k and B k represents the weight and bias value in hidden layer block k, y k represents the initial output, x k represents the hidden layer input, and y k′ represents the normalized output, and Among them, layer j is determined as its maximum absolute value weight (Amax j ) is smaller than the layer of GM; Scale the weight values ​​in layer j to in, Set P = P / M j ;as well as Propagate P to subsequent hidden layers until P = 1.

21. The method of claim 18, wherein: In step d), normalizing layer i includes: If the propagation normalization is only performed in the forward direction (j > i), then Otherwise, if the propagation normalization is only performed in the backward direction (j < i), then Among them, A i and B i represents the weight and bias value in hidden layer block i, y i represents the initial output, x i represents the hidden layer input, and y i′ Represents normalized output.

22. The method of claim 21, wherein: If the propagation normalization in the forward direction is followed by the propagation normalization in the backward direction, then layer i is renormalized to 23. A method for processing an encoded video sequence using one or more neural networks, the method comprising: Receiving high-level syntax that indicates enabling loop filtering using a neural network (NNLF) for decoding a current picture; Parsing the high-level syntax to extract a parameter indicating a recommended position of an adaptive loop filter (ALF); And Decoding the current picture based on the ALF position parameter to generate an output picture, wherein the ALF position parameter specifies one of the following: Performing ALF filtering after the NNLF; Performing ALF filtering before the NNLF; Replacing the ALF filtering with a convolutional neural network (CNN) located after the NNLF; and Performing ALF filtering before an enhanced NNLF with input ALF classification data.

24. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for causing one or more processors to perform the method according to any one of claims 1 to 23.

25. An apparatus comprising a processor and configured to perform any one of the methods described in claims 1 to 23.

Images