Learning transform coefficients

Abstract

A method of applying filtering is provided. The method includes providing a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY. The method includes providing a set of inputs, at least one input i of spatial size of XxY. The method includes, for the least one input i in the set of inputs: (i) forming AxB sub-blocks of size MxN from the input; (ii) determining at least two coefficients for at least one of the AxB sub-blocks based on the corresponding sub-block, with at least one of the at least two coefficients based on at least two input values; and (iii) re- arranging the at least two coefficients to form a re-sized input of size of AxBxC(i), where C(i) ≤ (M*N). The method includes applying the neural network using the re-sized input to generate an output.

Description

LEARNING TRANSFORM COEFFICIENTSTECHNICAL FIELD

[0001] This disclosure relates to video compression, and more particularly, to learning transform coefficients for such video compression.BACKGROUND

[0002] Video compression

[0003] Video is the dominant form of data traffic in today’s networks and is projected to still increase its share [1], One way to reduce the data traffic from video is compression. Here the source video is encoded to a bitstream, which then can be stored and transmitted to end users. Using a decoder, the end user can extract the video data and display it on a screen. However, since the encoder may not know what kind of device the encoded bitstream is going to be sent to, it has to compress the video to a predetermined format, such as the standardized format VVC. This way, all devices that support the chosen standard can decode the video. Compression can be lossless, i.e., the decoded video will be identical to the source given to the encoder, or lossy, where a certain degradation of content is accepted. Using lossy compression allows for significantly lower bit rates, i.e., the compression ratio can be much higher. This is because reproducing image noise perfectly can make lossless compression quite expensive.

[0004] A video sequence contains a sequence of pictures. A color space commonly used in video sequences is YCbCr, where Y is the luma (brightness) component and Cb and Cr are the chroma components. Sometimes the Cb and Cr components are called U and V. Other color spaces are also used, such as ICtCp, IPT, constant-luminance YCbCr, RGB, YCoCg, etc., and embodiments disclosed herein are applicable also in these cases. The order that the pictures are placed in in the video sequence when viewed is called ‘display order’. Each picture is assigned with a Picture Order Count (POC) value to indicate its position in terms of display order. In this document we interchangeably use the terms ‘images’, ‘pictures’ or ‘frames’.

[0005] Video compression is used to compress video sequences into a sequence of coded pictures. In many existing video codecs, the picture is divided into blocks of different sizes. A block is a two-dimensional array of samples. The blocks serve as the basis for coding. A video decoder then decodes the coded pictures into pictures containing sample values.

[0006] Commonly used video coding standards

[0007] Video standards are usually developed by international organizations as these represent different companies and research institutes with different areas of expertise and interests. Currently, the most applied video compression standard is H.264 / AVC which was jointly developed by ITU-T and ISO. The first version of H.264 / AVC was finalized in 2003, with several updates in the following years. The successor of H.264 / AVC, which was also developed by ITU-T and ISO, is known as H.265 / HEVC (High Efficiency Video Coding) and was finalized in 2013. MPEG and ITU-T have created a successor to HEVC within the Joint Video Exploratory Team (JVET). The name of this video codec is Versatile Video Coding (VVC) and version 1 of the VVC specification has been published as Rec. ITU-T H.266 | ISO / IEC 23090-3, “Versatile Video Coding”, 2020.

[0008] The VVC video coding standard is a block-based video codec and utilizes both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using uni-directional (P) or bi-directional inter (B) prediction at the block level from previously decoded reference pictures. In the encoder, the difference between the original pixel data and the predicted pixel data, referred to as the residual, is transformed into the frequency domain, quantized, and then entropy coded before transmitted together with necessary prediction parameters such as prediction mode and motion vectors, also entropy coded. The decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual, and then adds the residual to the intra or inter prediction to reconstruct a picture.

[0009] The VVC video coding standard uses a block structure referred to as quadtree plus binary tree plus ternary tree block structure (QTBT+TT) where each picture is first partitioned into square blocks called coding tree units (CTU). All CTUs are of the same size and the partitioning of the picture into CTUs is done without any syntax controlling it. Each CTU is further partitioned into coding units (CUs) that can have either square or rectangular shapes. The CTU is first partitioned by a quad tree structure, and then it may be further partitioned with equally sized partitions either vertically or horizontally in a binary structure to form coding units (CUs). A block could thus have either a square or rectangular shape. The depth of the quad tree and binary tree can be set by the encoder in the bitstream. The ternary tree (TT) part adds the possibility to divide a CU into three partitions instead of two equally sized partitions; this increases the possibilities to use a block structure thatbetter fits the content structure of a picture, such as roughly following important edges in the picture.

[0010] A block that is intra coded is an I-block. A block that is uni-directional predicted is a P-block and that is bi-directional predicted a B-block. For some blocks, the encoder decides that encoding the residual is not necessary, perhaps because the prediction is sufficiently close to the original. The encoder then signals to the decoder that the transform coding of that block should be bypassed, i.e., skipped. Such a block is referred to as a skip-block.

[0011] Transforms, such as discrete cosine transform (DCT) and discrete sine transform (DST), are commonly used in video coding to transform the residual samples from the pixel domain to the frequency / transform domain. The transform is used to compact the information into a few transform coefficients and allow a more efficient compression.There are different types of DCT and DST, e.g., DCT has eight types I to VIII and DST also has eight types I to VIII, among which DCT type II is the most common one used in compression. In VVC, DCT-II, DCT-VIII, and DST-VII are used.

[0012] In video coding, block-based two-dimensional transforms are used as primary transforms. That is, a picture or a frame is separated into blocks of different sizes. For a block of size MxN, one transform is applied in the horizontal direction with dimension M and another transform is applied in the vertical direction with dimension N. The two transforms can be of different sizes and / or transform types. The values of M and N are usually 2, 4, 8, 16, 32, or 64.

[0013] Low-frequency non-separable transform (LFNST) is another type of transform used in VVC. It is applied between the primary transform and the quantization at the encoder side and de-quantization and inverse primary transform at the decoder side. Since these transforms are non-separable, they are applied directly to a block of samples rather than dimension by dimension.

[0014] Exploration Experiment on Neural Network based Video Coding (NNVC)

[0015] At the 20th JVET meeting it was decided [2] to set up an exploration experiment (EE) on neural network-based (NN-based) video coding. The exploration experiment continued at the subsequent JVET meetings 21 through 29 with many tests: NN-based inloop filtering, NN-based post filtering, NN-based super resolution and NN-based intraprediction. The first two are of special interest to embodiments described herein and will be described further.

[0016] VVC contains three in-loop filters that are not based on neural networks: A deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter (ALF). The deblocking filter is used to remove block artifacts by smoothening discontinuities in horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS can have values 0, 1, and 2, where a larger value indicates a stronger filtering. The output of the deblocking filter is further processed by SAO, and the output of SAO is then processed by ALF. The output of the ALF filter can then be put into the display picture buffer (DPB), which is used for prediction of subsequently encoded (or decoded) pictures. Since the deblocking filter, SAO filter and ALF filter in this way influence the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loopfilters. This means that changes done by the loopfilter may influence not only the current picture but future pictures. It is possible for a decoder to further filter the picture in the DPB, but not store the filtered output in the DPB, but only send it to the display / decoded file. In contrast to loopfilters, such a filter is not influencing future predictions and is therefore classified as a post-processing filter, also known as a postfilter.

[0017] The contributions JVET-X0066 [3] and JVET-Y0143 [4] are two successive contributions that describe NN-based in-loop filtering. Both contributions use the same NN models for filtering. The NN-based in-loop filter is placed before SAO and ALF and the samples before the deblocking filter are used as input to the filter. The purpose of using the NN-based filter is to improve the quality of the reconstructed samples. Here it is helpful that the NN model is non-linear. While deblocking, SAO and ALF all contain non-linear elements such as conditions, and are thus not strictly linear, all three of them are based on linear filters. In contrast, a sufficiently big NN model can in principle leam any non-linear mapping and is therefore capable of representing a wider class of functions compared to deblocking, SAO and ALF. In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters. In a refined version of that work presented in the contribution JVET-AB0052, only two models are used: One for luma samples and another model for chroma samples. The use of NN filtering can be controlled (turned on or off) on ablock (CTU) level or a picture level. The encoder can further determine the strength of the filtering for the blocks / pictures where it is turned on.

[0018] The two NN models are convolutional neural networks. The terms “neural network”, “neural network model”, and “NN model” are used interchangeably. Using the model for luma samples as an example, the model from JVET-AB0052 has five inputs, the reconstructed samples of luma before deblocking (called ‘rec’), the prediction samples of luma (‘pred’), the BS information of luma (bs), the quantization parameter (qp) as well as information on whether a particular sample was intra-predicted, uni-predicted or bipredicted (called ‘IPB’). These five inputs first go through a convolutional layer and a parametric rectified linear unit (PReLU) layer separately, then they are concatenated and fused together to generate the signal y, as shown in FIG. 1. This figure shows the “head” of the neural network, which consists of the input, a fusion part and a transition part.

[0019] Each input is connected to a convolutional layer with a kernel size of 3x3, denoted “conv3x3” in the picture, with a multitude of output channels. As an example, in JVET- AB0052, 96 channels are used for each input, whereas in JVET-AC0126, more channels are used for the ‘rec’ input and fewer are used for the ‘QP’ input, since the ‘rec’ input typically carries more information about the signal than does the ‘QP’ input. The fuse part contains a “convlxl” layer which is a convolutional layer with kernel size 1x1. In both JVET- AB0052 and JVET-AC0126, this layer has 5*96=480 input channels and 96 output channels. The PReLUs make up the activation layer. “Unsqueeze, expand” is the dimension manipulation that expands qp, which in the experimental software NNVC is just a single number, into the same size as other inputs such as a 128x128- or 256x256-sample array. It should be noted that in many applications it may be the case that different samples inside the CTU will be associated with different qps. In this case the qp would effectively be of the same resolution as the other inputs (rec, pred, part and bs) and the ’’unsqueeze, expand”- layer would not be needed. Finally, “concat” means concatenation. ”2],” denotes downsampling with a factor of 2.

[0020] After the head, there are N sequential residual blocks of the same structure. In JVET-AB0052 and JVET-AC0126, N=8. The first residual block inputs y and outputs zO. The i-th residual block outputs zi, i=0,..,7. The structure of the residual block is depicted in FIG. 2. After eight residual blocks, the signal z7 is processed by a convolutional layer, aPReLU, another convolutional layer, pixel shuffling, and a final scaling to generate the output, as shown in FIG. 3.

[0021] The chroma model is different from the luma models. The biggest difference is that the chroma models take the reconstructed luma samples (‘rec’) as input.

[0022] The NN-based in-loop filters presented in JVET-X0066, JVET-AB0053, JVET- AB0052 and JVET-AC0126 increase the compression efficiency of the codec substantially, i.e., they lower the bit rate substantially without lowering the objective quality as measured by MSE-based PSNR. Increases in compression efficiency, often referred to simply as “gain”, are often measured as the Bjontegaard-delta rate (BDR) against an anchor. As an example, a BDR of -1% means that the same PSNR distortion can be reached with 1% fewer bits. As reported in JVET-Y0143, for the random access (RA) configuration, the BDR gain for the luma component (Y) is -9.80%, and for the all-intra (Al) configuration, the BDR gain for the luma component is -7.39%. The complexity of NN models used for compression are often measured by MAC / pixel (multiply-accumulate operations per pixel). The high gain of NN model is directly related to the high complexity of the NN model. The luma intra model described in JVET-Y0143 has a complexity of 430 kMAC / pixel, i.e., 430,000 multiply-accumulate operations per pixel. There are also other measures of complexity, such as total model size in terms of stored parameters.

[0023] References:

[0024] [1] P. Cerwall (executive editor), et al. Ericsson Mobility Report. https: / / www.ericsson.com / en / mobility-report. November 2019

[0025] [2] BoG Report: Neural Network Technology, A. Segall, JVET-T0130, October 2020.

[0026] [3] EE1-1.6: Combined Test of EE1-1.2 and EE1-1.4, Y. Li, K. Zhang, L. Zhang, H. Wang, J. Chen, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-X0066, Oct. 2021

[0027] [4] EE1-1.2: Test on Deep In-Loop Filter with Adaptive Parameter Selection and Residual Scaling, Y. Li, K. Zhang, L. Zhang, H. Wang, K. Reuze, A.M. Kotra, M. Karczewicz, JVET-Y0143, Jan. 2022SUMMARY

[0028] Complexity is a major challenge in existing NN models used for video coding.Methods for improving the complexity-performance tradeoff are highly valued in NN-based video coding. The structure of the NN model described in previous art can be improved. It is possible to keep the compression efficiency performance the same or even improve it while lowering the complexity. The high complexity of current NN models is a major challenge for practical hardware implementations. Reducing the complexity while preserving the NN compression efficiency performance is therefore highly desirable. As an example, a lower number of kMACs / sample means that less power is consumed in a wireless device, increasing battery life. Furthermore, a lower power consumption also means that the device does not become as hot. It also means that a smaller surface area of a chip needs to be devoted to video decoding, lowering manufacturing cost. Therefore, a better tradeoff between the complexity and gain is highly desirable.

[0029] Current state-of-the-art NN-based loopfilters are built on convolutional neural networks. The inputs to the NN are in the pixel domain, i.e., they are directly taken from the reconstruction, prediction, bs, IPB, or QP. Embodiment described herein: 1) transform the inputs from the pixel domain into frequency / transform domain; and 2) rearrange the transform coefficients into frequency channels. The NN is trained to leam the transform coefficients. The main idea is to take advantage of the frequency representations of the inputs and let the NN leam the frequencies. The transform coefficients are rearranged such that the coefficients of the same frequencies are put together, and coefficients of different frequencies are put in different channels. Embodiments can therefore provide an improved tradeoff between complexity and gain.

[0030] According to a first aspect, a method of applying filtering during video coding using a neural network is provided. The method includes providing a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y. The method includes providing a set of inputs, each input of spatial size of XxY. The method includes, for each input i in an input subset of the set of inputs: (i) forming (X / M)x(Y / N) sub-blocks of size M*N from the input, for integers M and N; (ii) determining (M*N) coefficients for each of the (X / M)x(Y / N) subblocks based on the corresponding (X / M)x(Y / N) sub-blocks; and (iii) re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input of size of(X / M)x(Y / N)xC(i), where C(z) < (M*N) and is a function of the input i. The method includes applying the neural network using the re-sized inputs for inputs in the input subset to generate an output.

[0031] According to a second aspect, a method of applying filtering during video coding using a neural network is provided. The method includes providing a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y. The method includes providing a set of inputs, each input of spatial size of XxY. The method includes applying the neural network using the set of inputs to generate an output. Applying the neural network comprises, for a first layer having an input signal x and second layer having an output signal y in the one or more additional layers of the neural network: (i) forming (X / M)x(Y / N) sub-blocks of size M*N from the input signal x, for integers M and N; (ii) determining (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks based on the corresponding (X / M)x(Y / N) sub-blocks; and (iii) re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a resized input signal x of size of (X / M)x(Y / N)x(M*N); (iv) feeding the re-sized input signal x into the first layer; and (v) re-arranging the output signal y of the second layer from a size of (X / M)x(Y / N)x(M*N) to a size of XxY.

[0032] According to a third aspect, a computer program is provided, comprising instructions which when executed by processing circuitry of a node, causes the node to perform the method of any of the embodiments of the first and second aspects.

[0033] According to a fourth aspect, a carrier is provided, containing the computer program of the third aspect. The carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0034] According to a fifth aspect, an encoder is provided. The encoder includes processing circuitry. The encoder includes a memory, the memory containing instructions executable by the processing circuitry. When the instructions are executed, the encoder is configured to perform the method of any one the embodiments of the first and second aspects.

[0035] According to a sixth aspect, a decoder is provided. The decoder includes processing circuitry. The decoder includes a memory, the memory containing instructions executable by the processing circuitry. When the instructions are executed, the encoder is configured to perform the method of any one the embodiments of the first and second aspects.BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0037] FIG. 1 illustrates part of a neural network (the “head”) for filtering a block of image samples according to related art.

[0038] FIG. 2 illustrates part of a neural network (the “body”) for filtering a block of image samples according to related art.

[0039] FIG. 3 illustrates part of a neural network (the “tail”) for filtering a block of image samples according to related art.

[0040] FIG. 4 is a chart illustrating the performance-complexity tradeoff according to an embodiment.

[0041] FIG. 5 illustrates part of a neural network (the “head”) for filtering a block of image samples according to an embodiment.

[0042] FIG. 6 illustrates part of a neural network (the “tail”) for filtering a block of image samples according to an embodiment.

[0043] FIG. 7 illustrates applying a transformation to some but not all of the channels of an input according to an embodiment.

[0044] FIG. 8 illustrates applying a 2x2 transformation according to an embodiment.

[0045] FIG. 9 illustrates part of a neural network (the “head”) for filtering a block of image samples according to an embodiment.

[0046] FIG. 10 illustrates a kernel according to an embodiment.

[0047] FIG. 11 illustrates a neural network for filtering a block of image samples according to an embodiment.

[0048] FIG. 12 illustrates a neural network for filtering a block of image samples according to an embodiment.

[0049] FIG. 13 is a flow chart according to an embodiment.

[0050] FIG. 14 is a flow chart according to an embodiment.

[0051] FIG. 15 is a block diagram of a node (e.g., an encoder or decoder) according to an embodiment.DETAILED DESCRIPTION

[0052] One advantage of embodiments disclosed herein is providing a better complexity performance tradeoff. Computational complexity for a NN may be expressed in kMAC / sample and compression efficiency performance may be expressed as a BD-rate gain. In other terms, using embodiments disclosed herein, the same or similar compression efficiency is achieved with lower complexity, or the same or similar complexity can achieve higher compression efficiency. This is further described below using some examples.

[0053] After the transform step is applied, different coefficients capture different frequency properties. Some coefficients may carry more information than others. The NN will leam the coefficients in the frequency domain, which can provide more flexibility in frequency manipulation. By rearranging the transform coefficients, the spatial resolution is reduced. This will reduce the complexity in terms of kMAC / sample and provide a promising tradeoff between the complexity and the gain.

[0054] The following explains how the complexity may be reduced. For example, assume an input has a size of 256x256x1. Applying a block-based 2x2 DCT transform and rearranging will transform the input into coefficients of size 128x128x4. The first conv3x3 layer is changed to adapt to the increased channel, so the number of input channels is increased from 1 to 4. The number of output channels does not change, so it is still 96. The complexity of the first conv3x3 is thus 128*128*3*3*4*96=56623104 MACs, and this is the same as the original complexity 256*256*3*3*1*96=56623104 MACs. The MAC / sample is computed using the original size, so divided by 256x256 it is 56623104 / 256 / 256=864 MAC / sample. There are several conv3x3 layers associated with the inputs, and the complexity for these convolutional layers remains the same.

[0055] The channels inside the network do not change, and there are 96 channels. Now, for each conv3x3 layer, the spatial dimension is 128x128 instead of 256x256, the complexity becomes 128*128*3*3*96*96, but we are still dividing by the original size 256x256 to obtain MAC / sample, which is 128*128*3*3*96*96 / 256 / 256=20 736 MAC / sample. However, the original complexity is 256*256*3*3*96*96 / 256 / 256= 82 944 MAC / sample. Comparing these two, the new complexity is only a quarter of the original complexity. Thereare multiple convolutional layers of this kind, and this contributes to the complexity reduction.

[0056] Similarly, if the convolutional layer has a kernel size 1x1, i.e., convlxl, the new complexity is 128*128*1*1*96*96 / 256 / 256=2304 MAC / sample and the original complexity is 256*256*1*1*96*96 / 256 / 256=9216 MAC / sample, which is also a factor of 4.

[0057] If there is a downsampling process, e.g., 2], in the transition block of FIG. 1, the spatial dimension is reduced to a quarter. The new complexity of a conv3x3 layer is (128 / 2)*(128 / 2)*3*3*96*96 / 256 / 256=5184 MAC / sample. The original complexity of a conv3x3 layer is (256 / 2)*(256 / 2)*3*3*96*96 / 256 / 256=20736 MAC / sample. This is still a factor of 4.

[0058] Adding the complexity of all the convolutional layers, the conv3x3 layers associated with inputs have unchanged complexity, but all other convolutional layers (convlxl and conv3x3) are of a quarter complexity. Thus, the final complexity of 128x128x4 is slightly larger than a quarter of the complexity of 256x256. As a result, after the transform and rearranging, the complexity is 110 kMAC / sample, while the original complexity is 425 kMAC / sample.

[0059] FIG. 4 shows the performance vs. complexity for some examples, the lower-left the better. For Random Access (RA) configuration (left), without using any NN (“bypass luma” in the figure), there is no gain. Using an NN with 8 residual blocks and a complexity of 425 kMAC / pix, the performance gives a BD-rate of -9.95% for luma. By simply reducing the number of residual blocks from 8 to 1, the complexity is 135 kMAC / sample, and the gain is - 5.62%. If a 2x2 DCT transform is applied to the inputs, we obtain a complexity of 110 kMAC / sample and a gain of -7.83%, which shows a better tradeoff than simply reducing the number of residual blocks. This is also observed for the All-Intra (Al) configuration (right).

[0060] Another comparison is to skip the transform step but still perform the rearranging step. With this, the input will still be rearranged into 4 channels with reduced spatial dimension. However, the 4 channels do not represent different frequencies as they are in the pixel domain. This gives the same complexity of 110 kMAC / sample as using the 2x2 transform. However, the performance is -7.57% for RA. When compared to performance of - 7.83% achieved by applying both transform and rearrangement, the 2x2 transform contributes to 0.26% performance improvement.

[0061] In one embodiment, transforms are applied to inputs before the convolutional layer “conv3x3”. One example of this embodiment is shown in FIGS. 5 and 6. FIG. 5 illustrates that the transforms are applied to the inputs rec, pred, bs, IPB, and / or QP of this neural network. The block “Trans” indicates that transforms are applied. FIG. 6 illustrates that the inverse transform “Inv-Trans” is applied after the NN output, to bring the coefficients back to the pixel domain.

[0062] In another embodiment, it is possible to use DCT with dimension 2x2, 4x4, 8x8, or MxN (M and N are integers and need not be equal). The transform is not limited to DCT type II. It can be DCT with other types, DST, or even wavelets, which can transform the input into different frequency bands. For the 2x2 sample case, the 2D DCT transform is equivalent to the 2D 2x2 Haar wavelet transform. For bigger transforms, such as 4x4, the 2D DCT is no longer equivalent to the 2D 4x4 Haar wavelet transform. For the case of 4x4 samples or larger, one embodiment of the present invention is to use the 2D DCT transform, another is to use the DST transform, another to use the 2D Haar wavelet transform, yet another one is to use another 2D wavelet transform, such as the one used in JPEG 2000.

[0063] In another embodiment, it is possible to apply transform to partial inputs, e.g., only the “rec” input in the example of FIG. 5, and combine the transform coefficients from “rec” with non-transformed inputs. In another example, if the QP is a constant value, one may skip the transform for the QP input and use only the mean / constant value. This is due to the fact that in some cases, the QP may be constant over the entire block and the transformed version may only have non-zero coefficients in the DC channel while all the higher frequency coefficients may be zero. In other cases the QP may not be constant over the entire block and transforming the QP input may be beneficial.

[0064] In another embodiment, if the input(s) include both luma samples and chroma samples, that is, the dimension is height x width x color channel, it is possible to apply transforms to all color channels or only one or some of the color channels.

[0065] For example, as shown in FIG. 7, if “rec” has a size of 256x256x3 which include luma reconstruction with 256x256x1 and chroma reconstruction with dimension 256x256x2 which is upsampled from original chroma dimension 128x128x2, applying a 2x2 transform is only meaningful to luma but not chroma, since the samples in every 2x2 block of the chroma are the same as it has been upsampled / duplicated, and it is fine to take the one sample in every2x2 block for chroma. In this case, we only need to apply transform to the luma channel of “rec”. As a result, luma gives 4 frequency channels, and chroma U and V gives 2 channels.

[0066] The steps for applying the 2x2 transform and rearranging the transform coefficients are shown in FIG. 8, assuming that the inputs to the neural network are denoted by XI to Xn and the size of each input is 256x256x1. Illustrated steps in FIG. 8 are described here in more details:• A 256x256x1 input is split into 128x128 number of 2x2 blocks.• A 2x2 transform is applied to each 2x2 block. This will give 4 transform coefficients representing 4 frequencies for each block.• The transform coefficients are rearranged into 128x128x4, where the coefficients representing the same frequency are put into one channel and so there are 4 channels.• The 128x128x4 transform coefficients of each input are fed into the NN.

[0067] If we train the NN to target the transform coefficients, the output of NN will be learned transform coefficients. Then, we can apply inverse transform to bring the output coefficients back into pixel domain.• The samples are rearranged from 128x128x4 to 256x256.• From the output of the NN, inverse 2x2 transform is applied to each 2x2 block.

[0068] Note that, the order of rearranging the samples and applying transforms does not matter, as long as the right coefficients are used.

[0069] For example, from the NN input side, if we first rearrange the samples, the input will be reshaped to 128x128x4. Then, we apply the 2x2 transform to each 1x1x4 coefficients for all 128x128 positions.

[0070] Similarly, from the NN output side, if we apply 2x2 transform first, we simply take 1x1x4 coefficients from 128x128x4 output. These four coefficients are inverse transformed. Apply this to all the 128x128 positions. Then, we can rearrange the inverse transformed 128x128x4 samples back to 256x256.

[0071] If the transform is 4x4, the rearranged transform coefficients will give a size of 64x64x16, where 16 is the channel number. If the transform is 8x8, the rearranged transform coefficients will give a size of 32x32x64, where 64 is the channel number.

[0072] In another embodiment, the implementation of the rearranging step and the inverserearranging step uses PyTorch functions PixelUnshuffle() and PixelShuffle(), respectively.

[0073] In yet another embodiment, we only rearrange the samples without transform. For example, a 256x256 input is split into 128x128 number of 2x2 blocks. We directly reshape the 2x2 block into 1x1x4 dimension and no transform is applied. Therefore, we can still get the input size of 128x128x4. This can take the advantage of lowering the complexity in kMAC / sample by reducing the spatial dimension, but it loses the advantage of separating channels into different frequencies.

[0074] Since different inputs have different frequency properties, one can use a subset of frequency channels to further reduce the complexity without impacting the performance. FIG. 9 shows the dimension of the transform coefficients after the 2x2 DCT transform, where Cl to C5 show the number of channels corresponding to each input after the 2x2 DCT transform and Cl to C5 are not necessarily equal. For example, for QP, one can use only the first frequency channel and discard the remaining channels, since the first frequency channel is the most important one. This will give C5=l and the dimension of QP transform coefficient is 128x128x1.

[0075] Similarly, reducing the channels in Cl to C4 is possible. However, keeping Cl with full size is probably important as it contains most of the information. On the other hand, it makes sense to reduce C3 and C4, for example, using only the two or three frequency channels for bs and IPB and discarding the remaining channels, since bs and IPB contain a lot of flat areas and they do not have much high frequencies. Thus, we can choose C3=2 or 3 and C4=2 or 3 and the dimension becomes 128x128x2 or 128x128x3 for those inputs.

[0076] In a similar fashion, it is also possible to treat the different frequency channels differently. As an example, after the transformation, the rec output has four channels. However, the first of these channels represents the average value and may thus be more important than the other three channels. Therefore, extra computational power may be spent on that channel. In one embodiment, the first channel therefore gets a bigger kernel, e.g., 5x5, and the other three channels gets a smaller kernel, e.g., 3x3. This can be trained by giving all four channels a 5x5 kernel but forcing all the kernel weights closest to the edge to zero. This is shown in FIG. 10, where a full 5x5 kernel (1001) is used for the first channel, but for the remaining three channels, a 5x5 kernel (1002) is used where all the weights marked with a hatched pattern (1003) are forced to zero during training and only the remaining weights(1004) are allowed to take non-zero values. During employment of the neural network, it is possible to run compression and decompression with only a 3x3 kernel for the last three channels, lowering computational complexity.

[0077] In another embodiment, the transform is applied not in the input layer but in other layer(s) furthur in the body of the neural network. In this embodiment the inverse transform may be applied after the NN output or in another layer before the output layer and after the layer where transform has been applied. In one example the transform is applied before a convolutional layer, e.g.“conv3x3” in the body of the NN and an inverse transform is applied after the said convolutional layer. FIG. 11 illustrates one example of this embodiment.

[0078] Fig. 11. Transform is applied in a middle layer of the neural network

[0079] In another embodiment, there is an internal branch in the neural network which separates from the main branch of the NN and its output joins the main branch after one or more layers, and the transform and inverse transform are applied only to that branch or some of the channels in that branch. FIG. 12 illustrates one example of this embodiment.

[0080] In the above description, AxBxC expresses a size of AxB with C channels and AxBxl is the same as AxB.

[0081] FIG. 13 illustrates a flow chart according to an embodiment. Process 1300 is a method of applying filtering during video encoding and / or decoding using a neural network. The method may begin with step si 302.

[0082] Step si 302 comprises providing a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y.

[0083] Step si 304 comprises providing a set of inputs, at least one input i of spatial size of XxY.

[0084] Step sl306 comprises, for the least one input i in the set of inputs (s 1306):(i) forming AxB sub-blocks of size MxN from the input, for integers A, B, M, and N (step 1308);(ii) determining at least two coefficients for at least one of the AxB sub-blocks based on the corresponding sub-block, wherein at least one of the at least twocoefficients is based on at least two input values in the set of inputs (step S1310); and(iii) re-arranging the at least two coefficients for at least one of the AxB subblocks to form a re-sized input of size of AxBxC(i), where C(i) < (M*N) (step S1312).

[0085] Step si 314 comprises applying the neural network using the at least one re-sized input in the set of inputs to generate an output.

[0086] In some embodiments, for at least one input i in the set of inputs, C(i) = (M*N) such that all the (M*N) coefficients are used in the re-sized input. In some embodiments, for at least one input i in the set of inputs, C(i) < (M*N). In some embodiments, the convolutional layer of the neural network receives the at least one re-sized input in the set of inputs, and wherein a kernel in the convolutional layer of one of the C(i) channels belonging to the input i has a kernel size that is the same size as each other kernel in the convolutional layer for each other C(i) channel. In some embodiments, the convolutional layer of the neural network receives the at least one re-sized input in the set of inputs, and wherein a kernel in the convolutional layer of one of the C(i) channels belonging to the input i has a kernel size that is different from at least one other kernel in the convolutional layer for at least one other C(i) channel.

[0087] In some embodiments, the block of image samples and at least one input in the set of inputs have a third non-spatial dimension of size Z > 1, and the steps (i) - (iii) are performed for at least one input i in the set of inputs and for one or more channels c of the input i, where the channels c correspond to the third non-spatial dimension of size Z. In some embodiments, the method further includes re-arranging the output from a size of (X / M)x(Y / N)x(M*N) to a size of XxY. In some embodiments, the method further includes re-arranging the output from a size of A'xB'xC'(i), where C'(i) < (M*N), to a single channel output. In some embodiments, for at least one input i in the set of inputs, re-arranging the at least two coefficients for at least one of the AxB sub-blocks to form a re-sized input comprises using a PixelUnshuffle() function. In some embodiments, re-arranging the output comprises using a PixelShuffle() function.

[0088] In some embodiments, for at least one input i in the set of inputs, determining the at least two coefficients for at least one of the AxB sub-blocks comprises applying a frequency-domain transformation of size MxN to the corresponding AxB sub-block, resulting in the at least two coefficients belonging to one of (M*N) frequency bands. In some embodiments, rearranging the at least two coefficients for at least one of the AxB sub-blocks to form a resized input of size of AxBxC(i) comprises putting the at least two coefficients for at least one of the AxB sub-blocks into channels based on a frequency band of the (M*N) frequency bands that the at least two coefficients belong to. In some embodiments, the method further includes applying an inverse transformation of size MxN to the output of the neural network, wherein the inverse transformation is an inverse operation to the frequency-domain transformation.

[0089] In some embodiments, the set of inputs includes one or more of: (i) reconstructed samples before deblocking, (ii) prediction samples, (iii) block boundary strength information, (iv) a quantization parameter, and (v) information on whether a particular sample was intrapredicted, uni-predicted, or bi-predicted. In some embodiments, the frequency-domain transformation comprises one of a discrete cosine transform, a discrete sine transform, and a discrete wavelet transform. In some embodiments, both M and N equal one of 2, 4, and 8. In some embodiments, the method is applied during an in-loop filter. In some embodiments, the method is applied during a post-processing filter.

[0090] FIG. 14 illustrates a flow chart according to an embodiment. Process 1400 is a method of applying filtering during video encoding and / or decoding using a neural network. The method may begin with step S1402.

[0091] Step s!402 comprises providing a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y.

[0092] Step si 404 comprises providing a set of inputs, at least one input of spatial size of XxY.

[0093] Step si 406 comprises applying the neural network using the set of inputs to generate an output. Applying the neural network comprises, for a first layer having an input signal x and second layer having an output signal y in the one or more additional layers of the neural network:(i) forming AxB sub-blocks of size MxN from the input signal x, for integers A,B, M, and N (step S1408);(ii) determining at least two coefficients for at least one of the AxB sub-blocks based on the corresponding sub-block, wherein at least one of the at least two coefficients is based on at least two input values from the input signal x (step S1410); and(iii) re-arranging the at least two coefficients for the at least one AxB subblocks to form a re-sized input signal x of size of AxBx(M*N) (step S1412); and(iv) feeding the re-sized input signal x into the first layer (step S1414); and(v) re-arranging the output signal y of the second layer from a size of A'xB'xC'(i), where C'(i) < (M*N), to a single channel output (step S1416).

[0094] In some embodiments, re-arranging the at least two coefficients for at least one of the AxB sub-blocks to form a re-sized input signal x comprises using a PixelUnshuffie() function. In some embodiments, re-arranging the output signal y of the second layer comprises using a PixelShuffie() function. In some embodiments, the block of image samples and at least one input in the set of inputs have a third non-spatial dimension of size Z > 1, and the steps (i) - (v) are performed for one or more channels c of the block of image samples, where the channels c correspond to the third non-spatial dimension of size Z. In some embodiments, the one or more additional layers include a third layer and a fourth layer, and wherein the layers between and including the first layer and the second layer constitute a branch in the neural network, such that the input signal x is fed into the third layer and the output signal y joins with the output of the fourth layer.

[0095] In some embodiments, (ii) determining the at least two coefficients for at least one of the AxB sub-blocks comprises applying a frequency-domain transformation of size MxN to the corresponding AxB sub-block, resulting in the at least two coefficients belonging to one of (M*N) frequency bands. In some embodiments, (iii) re-arranging the at least two coefficients for the at least one AxB sub-blocks to form a re-sized input signal x of size of AxBxC(i) comprises putting the at least two coefficients for at least one of the AxB subblocks into channels based on a frequency band of the (M*N) frequency bands that the at least two coefficients belong to. In some embodiments, the method further includes applying an inverse transformation of size MxN to the output signal y, wherein the inverse transformation is an inverse operation to the frequency-domain transformation.

[0096] In some embodiments, the set of inputs includes one or more of: (i) reconstructed samples before deblocking, (ii) prediction samples, (iii) block boundary strength information, (iv) a quantization parameter, and (v) information on whether a particular sample was intrapredicted, uni-predicted, or bi-predicted. In some embodiments, the frequency-domain transformation comprises one of a discrete cosine transform, a discrete sine transform, and a discrete wavelet transform. In some embodiments, both M and N equal one of 2, 4, and 8. In some embodiments, the method is applied during an in-loop filter. In some embodiments, the method is applied during a post-processing filter.

[0097] FIG. 15 is a block diagram of node 1500 (e.g., an encoder or decoder), according to some embodiments. As shown in FIG. 15, node 1500 may comprise: processing circuitry (PC) 702, which may include one or more processors (P) 1555 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., node 1500 may be a distributed computing apparatus); at least one network interface 1548 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 1545 and a receiver (Rx) 1547 for enabling node 1500 to transmit data to and receive data from other nodes connected to a network 1510 (e.g., an Internet Protocol (IP) network) to which network interface 1548 is connected (physically or wirelessly) (e.g., network interface 1548 may be coupled to an antenna arrangement comprising one or more antennas for enabling node 1500 to wirelessly transmit / receive data); and a local storage unit (a.k.a., “data storage system”) 1508, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1502 includes a programmable processor, a computer program product (CPP) 1541 may be provided. CPP 1541 includes a computer readable medium (CRM) 1542 storing a computer program (CP) 1543 comprising computer readable instructions (CRI) 1544. CRM 1542 may be anon- transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1544 of computer program 1543 is configured such that when executed by PC 1502, the CRI causes node 1500 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, node 1500 may be configured to perform steps described herein without the need for code. That is, for example,PC 1502 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0098] As used herein, a network element, node, or subsystem (e.g., an encoder or decoder) may be comprised of one or more pieces of service network equipment, including hardware and software that communicatively interconnects other equipment on a network (e.g., other network elements, end stations, etc.), and is adapted to host one or more applications or services, either in a virtualized / non- virtualized environment, with respect to a plurality of subscribers and associated user equipment (UE) nodes that are operative to receive / consume content in a media distribution network where media content assets may be distributed and delivered using stream-based or file-based mechanisms. As such, some network elements may be disposed in a wireless radio network environment whereas other network elements may be disposed in a public packet-switched network infrastructure, including or otherwise involving suitable content delivery network (CDN) infrastructure that may comprise public, private, or mixed CDNs. Further, suitable network elements including one or more embodiments set forth herein may involve terrestrial and / or satellite broadband delivery infrastructures, e.g., a Digital Subscriber Line (DSL) network architecture, a Data Over Cable Service Interface Specification (DOCSIS)-compliant Cable Modem Termination System (CMTS) architecture, switched digital video (SDV) network architecture, a Hybrid Fiber- Coaxial (HFC) network architecture, a suitable satellite access network architecture or a broadband wireless access network architecture over cellular and / or WiFi connectivity. Accordingly, some network elements may comprise “multiple services network elements” that provide support for multiple network-based functions (e.g., 360° immersive A / V media preparation delivery policy management, session control, QoS policy enforcement, bandwidth scheduling management, content provider priority policy management, streaming policy management, and the like), in addition to providing support for multiple application services (e.g., data and multimedia applications including 360° immersive video assets (also referred to as 360-degree video assets or simply 360 video assets) in varying qualities or definitions). Example subscriber end stations or client devices may comprise various devices, tethered or untethered, that may consume or deliver media content assets using streaming and / or file-based downloading technologies, which may involve some type of rate adaptation in certain embodiments. Illustrative client devices or UE devices may therefore include any device configured to execute, inter aha, one or more client applications for receiving, recording, storing, and / or decoding / rendering 360 video content, live media and / orstatic / on-demand media, which may comprise Virtual Reality (VR) media, Augmented Reality (AR) media, Mixed Reality (MR) media, from one or more content providers, e.g., via a broadband access network, using HTTP, HTTPS, RTP, and the like. Accordingly, such client devices may include Next Generation IP-based STBs, networked TVs, personal / digital video recorders (PVR / DVRs), networked media projectors, portable laptops, netbooks, palm tops, tablets, smartphones, multimedia / video phones, mobile / wireless user equipment, portable media players, portable gaming systems or consoles (such as the Wii®, Play Station 3®, etc.) operating in concert with 3D display devices and the like, which may access or consume 360-degree content / services provided via a suitable media distribution network wherein a bandwidth and Quality of Experience (QoE) scheme may be provided in accordance with to one or more embodiments set forth herein.

[0099] One or more embodiments of the present patent disclosure may be implemented using different combinations of software, firmware, and / or hardware. Thus, one or more of the techniques shown in the Figures (e.g., flowcharts) may be implemented using code and data stored and executed on one or more electronic devices or nodes (e.g., a subscriber client device or end station, a network element, etc.). Such electronic devices may store and communicate (internally and / or with other electronic devices over a network) code and data using computer-readable media, such as non-transitory computer-readable storage media (e.g., magnetic disks, optical disks, random access memory, read-only memory, flash memory devices, phase-change memory, etc.), transitory computer-readable transmission media (e.g., electrical, optical, acoustical or other form of propagated signals - such as carrier waves, infrared signals, digital signals), etc. In addition, such network elements may typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (e.g., non-transitory machine-readable storage media) as well as storage database(s), user input / output devices (e.g., a keyboard, a touch screen, a pointing device, and / or a display), and network connections for effectuating signaling and / or bearer media transmission. The coupling of the set of processors and other components may be typically through one or more buses and bridges (also termed as bus controllers), arranged in any known (e.g., symmetric / shared multiprocessing) or heretofore unknown architectures. Thus, the storage device or component of a given electronic device or network element may be configured to store code and / or data for execution on one or more processors of that element, node or electronic device for purposes of implementing one or more techniques of the present disclosure.

[0100] Skilled artisans will recognize that the foregoing generalized example network environment may be implemented in a hierarchical network architecture, with various aspects of media capture and preparation, including, e.g., source stream stitching, projection mapping, source media compression, tiled / ABR encoding / transcoding, packaging, etc., as well as distributing / uploading and edge node processes taking place in different network portions disposed at different hierarchical levels, involving one or more operators, content delivery networks (CDNs), edge networks, and the like. Further, in some implementations, at least some of the foregoing apparatuses and processes may be cloud-based. In some arrangements, a CDN can be a large distributed system of servers deployed in multiple data centers connected to the Internet or other public / private communications network. A CDN can be a managed or unmanaged network, and can also be a federation of managed or unmanaged networks.

[0101] An example embodiment of a media server / source system operatively associated within the foregoing example network environment may therefore be configured, e.g., as a global headend, to accept media content from live sources and / or static file sources, e.g., online content providers such as Hulu®, Netflix®, YouTube®, or Amazon® Prime, as well as VOD catalog or content providers or studios such as, e.g., Disney, Warner, Sony, etc. Media content from live sources may comprise live programming captured relative to any type of event, e.g., sporting / entertainment / gaming events, concerts, live TV shows, live news broadcasting sources, such as, for instance, national broadcasters (e.g., NBC, ABC, etc.) as well as cable broadcaster channels like Time Warner channels of CNN, ESPN, CNBC, etc., and local broadcasters, etc., including any secondary media insertions such as advertisement media channels.

[0102] Summary of Various EmbodimentsAl. A method of applying filtering during video coding using a neural network, the method comprising: providing a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; providing a set of inputs, each input of spatial size of XxY; for each input i in an input subset of the set of inputs:(i) forming (X / M)x(Y / N) sub-blocks of size M*N from the input, for integers M and N;(ii) determining (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks based on the corresponding (X / M)x(Y / N) sub-blocks; and(iii) re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input of size of (X / M)x(Y / N)xC(z), where C(z) < (M*N) and is a function of the input z; and applying the neural network using the re-sized inputs for inputs in the input subset to generate an output.Ala. The method of embodiment Al, wherein for each input z in the input subset of the set of inputs, C(z) = (M*N) such that all the (M*N) coefficients are used in the re-sized input.Alb. The method of embodiment Al, wherein for one or more inputs z in the input subset of the set of inputs, C(z) < (M*N) such that some of the (M*N) coefficients are not used in the re-sized input.Ale. The method of any one of embodiment Al, Ala, and Alb, wherein the convolutional layer of the neural network receives the re-sized inputs for inputs in the input subset, and wherein a kernel in the convolutional layer for each of the C(z) channels belonging to the input z has a kernel size that is a function of the corresponding channel.Aid. The method of any one of embodiment Al, Ala, Alb, and Ale, wherein the block of image samples and each input of the set of inputs have a third non-spatial dimension of size Z > 1, and the steps (i) - (iii) are performed for each input z in the input subset of the set of inputs and for one or more channels c of the input z, where the channels c correspond to the third non-spatial dimension of size Z.A2. The method of any one of embodiments Al, Ala, Alb, Ale, and Aid, further comprising re-arranging the output from a size of (X / M)x(Y / N)x(M*N) to a size of XxY.A2a. The method of any one of embodiments A1-A2, wherein for each input z in an input subset of the set of inputs, re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input comprises using a PyTorch PixelUnshuffleO function.A2b. The method of any one of embodiments A2 and A2a, wherein re-arranging the output from a size of (X / M)x(Y / N)x(M*N) to a size of XxY comprises using a PyTorch PixelShuffleQ function.A3. The method of any one of embodiments Al-A2b, wherein, for each input i in an input subset of the set of inputs, determining the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks comprises applying a frequency-domain transformation of size MxN to the corresponding (X / M)x(Y / N) sub-block, resulting in each of the (M*N) coefficients belonging to one of (M*N) frequency bands; and re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input of size of (X / M)x(Y / N)xC(z) comprises putting the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks into channels based on a frequency band of the (M*N) frequency bands that each of the (M*N) coefficient belongs to.A4. The method of embodiment A3, further comprising applying an inverse transformation of size M*N to the output of the neural network, wherein the inverse transformation is an inverse operation to the frequency-domain transformation.A5. The method of any one of embodiments A1-A4, wherein the input subset of the set of inputs includes all of the set of inputs.A6. The method of any one of embodiments A1-A4, wherein the input subset of the set of inputs includes less than all of the set of inputs.A7. The method of any one of embodiments A1-A6, wherein the set of inputs includes one or more of: (i) reconstructed samples before deblocking, (ii) prediction samples, (iii) block boundary strength information, (iv) a quantization parameter, and (v) information on whether a particular sample was intra-predicted, uni-predicted, or bi-predicted.A8. The method of any one of embodiments A3-A4, wherein the frequency -domain transformation comprises one of a discrete cosine transform, a discrete sine transform, and a discrete wavelet transform.A9. The method of any one of embodiments A1-A8, wherein both M and N equal one of 2, 4, and 8.A10. The method of any one of embodiments A1-A9, wherein the method is applied during an in-loop filter.All. The method of any one of embodiments A1-A9, wherein the method is applied during a post-processing filter.Bl. A method of applying filtering during video coding using a neural network, the method comprising: providing a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; providing a set of inputs, each input of spatial size of XxY; applying the neural network using the set of inputs to generate an output, wherein applying the neural network comprises, for a first layer having an input signal x and second layer having an output signal y in the one or more additional layers of the neural network:(i) forming (X / M)x(Y / N) sub-blocks of size M*N from the input signal x, for integers M and N;(ii) determining (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks based on the corresponding (X / M)x(Y / N) sub-blocks; and(iii) re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input signal x of size of (X / M)x(Y / N)x(M*N); and(iv) feeding the re-sized input signal x into the first layer; and(v) re-arranging the output signal y of the second layer from a size of (X / M)x(Y / N)x(M*N) to a size of XxY.Bia. The method of embodiment Bl, wherein re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input signal x comprises using a PyTorch PixelUnshuffleQ function.Bib. The method of any one of embodiments Bl and Bia, wherein re-arranging the output signal y of the second layer from a size of (X / M)x(Y / N)x(M*N) to a size of XxY comprises using a PyTorch PixelShuffle() function.Bic. The method of any one of embodiment Bl, Bia, and Bib, wherein the block of image samples and each input of the set of inputs have a third non-spatial dimension of size Z > 1, and the steps (i) - (v) are performed for one or more channels c of the block of image samples, where the channels c correspond to the third non-spatial dimension of size Z.B2. The method of any one of embodiments Bl, Bia, Bib, and Bic, wherein the one or more additional layers include a third layer and a fourth layer, and wherein the layers between and including the first layer and the second layer constitute a branch in the neural network, such that the input signal x is fed into the third layer and the output signal y joins with the output of the fourth layer.B3. The method of any one of embodiments B1-B2, wherein:(ii) determining the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks comprises applying a frequency-domain transformation of size MxN to the corresponding (X / M)x(Y / N) sub-block, resulting in each of the (M*N) coefficients belonging to one of (M*N) frequency bands; and(iii) re-arranging the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input of size of (X / M)x(Y / N)x(M*N) comprises putting the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks into channels based on a frequency band of the (M*N) frequency bands that each of the (M*N) coefficient belongs to.B4. The method of embodiment B3, further comprising applying an inverse transformation of size M*N to the output signal , wherein the inverse transformation is an inverse operation to the frequency-domain transformation.B5. The method of any one of embodiments B1-B4, wherein the set of inputs includes one or more of: (i) reconstructed samples before deblocking, (ii) prediction samples, (iii) block boundary strength information, (iv) a quantization parameter, and (v) information on whether a particular sample was intra-predicted, uni-predicted, or bi-predicted.B6. The method of any one of embodiments B3-B4, wherein the frequency -domain transformation comprises one of a discrete cosine transform, a discrete sine transform, and a discrete wavelet transform.B7. The method of any one of embodiments B1-B6, wherein both M and N equal one of 2, 4, and 8.B8. The method of any one of embodiments B1-B7, wherein the method is applied during an in-loop filter.B9. The method of any one of embodiments B1-B7, wherein the method is applied during a post-processing filter.Cl. A computer program comprising instructions which when executed by processing circuitry of a node, causes the node to perform the method of any one of embodiments Al- Al l, and Bl-B9.C2. A carrier containing the computer program of embodiment Cl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.DI. An encoder, the encoder comprising: processing circuitry; and a memory, the memory containing instructions executable by the processing circuitry, whereby the encoder is configured to perform the method of any one the embodiments Al- Al l, and Bl-B9.D2. A decoder, the decoder comprising: processing circuitry; and a memory, the memory containing instructions executable by the processing circuitry, whereby the decoder is configured to perform the method of any one the embodiments Al- Al l, and Bl-B9.El. An encoder configured to apply filtering during video coding using a neural network, the encoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; provide a set of inputs, each input of spatial size of XxY ; for each input i in an input subset of the set of inputs:(i) form (X / M)x(Y / N) sub-blocks of size M*N from the input, for integers M and N;(ii) determine (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks based on the corresponding (X / M)x(Y / N) sub-blocks; and(iii) re-arrange the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input of size of (X / M)x(Y / N)xC(i), where C(z) < (M*N) and is a function of the input z; and apply the neural network using the re-sized inputs for inputs in the input subset to generate an output.E2. The encoder of embodiment El, wherein the encoder is further configured to perform the method of any one of embodiments A2-A11.Fl. An encoder configured to apply filtering during video coding using a neural network, the encoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; provide a set of inputs, each input of spatial size of XxY ; apply the neural network using the set of inputs to generate an output, wherein applying the neural network comprises, for a first layer having an input signal x and second layer having an output signal y in the one or more additional layers of the neural network:(i) form (X / M)x(Y / N) sub-blocks of size M*N from the input signal x, for integers M and N;(ii) determine (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks based on the corresponding (X / M)x(Y / N) sub-blocks; and(iii) re-arrange the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input signal x of size of (X / M)x(Y / N)x(M*N); and(iv) feed the re-sized input signal x into the first layer; and(v) re-arrange the output signal y of the second layer from a size of (X / M)x(Y / N)x(M*N) to a size of XxY.F2. The encoder of embodiment Fl, wherein the encoder is further configured to perform the method of any one of embodiments B2-B9.Gl. A decoder configured to apply filtering during video coding using a neural network, the decoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; provide a set of inputs, each input of spatial size of XxY ; for each input i in an input subset of the set of inputs:(i) form (X / M)x(Y / N) sub-blocks of size M*N from the input, for integers M and N;(ii) determine (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks based on the corresponding (X / M)x(Y / N) sub-blocks; and(iii) re-arrange the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input of size of (X / M)x(Y / N)xC(i), where C(z) < (M*N) and is a function of the input z; and apply the neural network using the re-sized inputs for inputs in the input subset to generate an output.G2. The decoder of embodiment Gl, wherein the decoder is further configured to perform the method of any one of embodiments A2-A11.Hl. A decoder configured to apply filtering during video coding using a neural network, the decoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; provide a set of inputs, each input of spatial size of XxY ; apply the neural network using the set of inputs to generate an output, wherein applying the neural network comprises, for a first layer having an input signal x and second layer having an output signal y in the one or more additional layers of the neural network:(i) form (X / M)x(Y / N) sub-blocks of size M*N from the input signal x, for integers M and N;(ii) determine (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks based on the corresponding (X / M)x(Y / N) sub-blocks; and(iii) re-arrange the (M*N) coefficients for each of the (X / M)x(Y / N) sub-blocks to form a re-sized input signal x of size of (X / M)x(Y / N)x(M*N); and(iv) feed the re-sized input signal x into the first layer; and(v) re-arrange the output signal y of the second layer from a size of (X / M)x(Y / N)x(M*N) to a size of XxY.H2. The decoder of embodiment Hl, wherein the decoder is further configured to perform the method of any one of embodiments B2-B9.

[0103] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0104] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method of applying filtering during video encoding and / or decoding using a neural network, the method comprising: providing (si 302) a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; providing (sl304) a set of inputs, at least one input i of spatial size of XxY; for the least one input i in the set of inputs (si 306):(i) forming (si 308) AxB sub-blocks of size MxN from the input, for integers A, B, M, and N;(ii) determining (si 310) at least two coefficients for at least one of the AxB subblocks based on the corresponding sub-block, wherein at least one of the at least two coefficients is based on at least two input values in the set of inputs; and(iii) re-arranging (si 312) the at least two coefficients for at least one of the AxB subblocks to form a re-sized input of size of AxBxC(i), where C(i) < (M*N); and applying (s 1314) the neural network using the at least one re-sized input in the set of inputs to generate an output.

2. The method of claim 1, wherein for at least one input i in the set of inputs, C(z) = (M*N) such that all the (M*N) coefficients are used in the re-sized input.

3. The method of claim 1, wherein for at least one input i in the set of inputs, C(z) < (M*N).

4. The method of any one of claims 1-3, wherein the convolutional layer of the neural network receives the at least one re-sized input in the set of inputs, and wherein a kernel in the convolutional layer of one of the C(z) channels belonging to the input z has a kernel size that is the same size as each other kernel in the convolutional layer for each other C(z) channel.

5. The method of any one of claims 1-3, wherein the convolutional layer of the neural network receives the at least one re-sized input in the set of inputs, and wherein a kernel in the convolutional layer of one of the C(z) channels belonging to the input i has a kernel size that is different from at least one other kernel in the convolutional layer for at least one other C(z) channel.

6. The method of any one of claims 1-5, wherein the block of image samples and at least one input in the set of inputs have a third non-spatial dimension of size Z > 1, and the steps (i) - (iii) are performed for at least one input i in the set of inputs and for one or more channels c of the input z, where the channels c correspond to the third non-spatial dimension of size Z.

7. The method of any one of claims 1-6, further comprising re-arranging the output from a size of (X / M)x(Y / N)x(M*N) to a size of XxY.

8. The method of any one of claims 1-7, further comprising re-arranging the output from a size of A'xB'xC'(i), where C'(i) < (M*N), to a single channel output.

9. The method of any one of claims 1-8, wherein for at least one input i in the set of inputs, re-arranging the at least two coefficients for at least one of the AxB sub-blocks to form a re-sized input comprises using a PixelUnshuffle() function.

10. The method of any one of claims 7-9, wherein re-arranging the output comprises using a PixelShuffle() function.

11. The method of any one of claims 1-10, wherein, for at least one input i in the set of inputs, determining the at least two coefficients for at least one of the AxB sub-blocks comprises applying a frequency-domain transformation of size MxN to the corresponding AxB sub-block, resulting in the at least two coefficients belonging to one of (M*N) frequency bands; and re-arranging the at least two coefficients for at least one of the AxB sub-blocks to form a re-sized input of size of AxBxC(i) comprises putting the at least two coefficients for atleast one of the AxB sub-blocks into channels based on a frequency band of the (M*N) frequency bands that the at least two coefficients belong to.

12. The method of claim 11, further comprising applying an inverse transformation of size MxN to the output of the neural network, wherein the inverse transformation is an inverse operation to the frequency-domain transformation.

13. The method of any one of embodiments 1-12, wherein the set of inputs includes one or more of: (i) reconstructed samples before deblocking, (ii) prediction samples, (iii) block boundary strength information, (iv) a quantization parameter, and (v) information on whether a particular sample was intra-predicted, uni-predicted, or bi-predicted.

14. The method of any one of claims 11-12, wherein the frequency -domain transformation comprises one of a discrete cosine transform, a discrete sine transform, and a discrete wavelet transform.

15. The method of any one of claims 1-14, wherein both M and N equal one of 2, 4, and 8.

16. The method of any one of claims 1-15, wherein the method is applied during an inloop filter.

17. The method of any one of claims 1-15, wherein the method is applied during a postprocessing filter.

18. A method of applying filtering during video encoding and / or decoding using a neural network, the method comprising: providing (si 402) a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; providing (sl404) a set of inputs, at least one input of spatial size of XxY; applying (si 406) the neural network using the set of inputs to generate an output, wherein applying the neural network comprises, for a first layer having an input signal x andsecond layer having an output signal y in the one or more additional layers of the neural network:(i) forming (si 408) AxB sub-blocks of size MxN from the input signal x, for integers A,B, M, and N;(ii) determining (sl410) at least two coefficients for at least one of the AxB subblocks based on the corresponding sub-block, wherein at least one of the at least two coefficients is based on at least two input values from the input signal x; and(iii) re-arranging (sl412) the at least two coefficients for the at least one AxB subblocks to form a re-sized input signal x of size of AxBx(M*N); and(iv) feeding (sl414) the re-sized input signal x into the first layer; and(v) re-arranging (sl416) the output signal y of the second layer from a size of A'xB'xC'(i), where C'(i) < (M*N), to a single channel output.

19. The method of claim 18, wherein re-arranging the at least two coefficients for at least one of the AxB sub-blocks to form a re-sized input signal x comprises using a PixelUnshuffleO function.

20. The method of any one of claims 18-19, wherein re-arranging the output signal y of the second layer comprises using a PixelShuffle() function.

21. The method of any one of claims 18-20, wherein the block of image samples and at least one input in the set of inputs have a third non-spatial dimension of size Z > 1, and the steps (i) - (v) are performed for one or more channels c of the block of image samples, where the channels c correspond to the third non-spatial dimension of size Z.

22. The method of any one of claims 18-21, wherein the one or more additional layers include a third layer and a fourth layer, and wherein the layers between and including the first layer and the second layer constitute a branch in the neural network, such that the input signal x is fed into the third layer and the output signal y joins with the output of the fourth layer.

23. The method of any one of claims 18-22, wherein:(ii) determining the at least two coefficients for at least one of the AxB sub-blocks comprises applying a frequency-domain transformation of size MxN to the correspondingAxB sub-block, resulting in the at least two coefficients belonging to one of (M*N) frequency bands; and(iii) re-arranging the at least two coefficients for the at least one AxB sub-blocks to form a re-sized input signal x of size of AxBxC(i) comprises putting the at least two coefficients for at least one of the AxB sub-blocks into channels based on a frequency band of the (M*N) frequency bands that the at least two coefficients belong to.

24. The method of claim 23, further comprising applying an inverse transformation of size MxN to the output signal , wherein the inverse transformation is an inverse operation to the frequency-domain transformation.

25. The method of any one of claims 18-24, wherein the set of inputs includes one or more of: (i) reconstructed samples before deblocking, (ii) prediction samples, (iii) block boundary strength information, (iv) a quantization parameter, and (v) information on whether a particular sample was intra-predicted, uni-predicted, or bi-predicted.

26. The method of any one of claims 23-24, wherein the frequency -domain transformation comprises one of a discrete cosine transform, a discrete sine transform, and a discrete wavelet transform.

27. The method of any one of claims 18-26, wherein both M and N equal one of 2, 4, and 8.

28. The method of any one of claims 18-27, wherein the method is applied during an inloop filter.

29. The method of any one of claims 18-27, wherein the method is applied during a postprocessing filter.

30. An encoder configured to apply filtering during video coding using a neural network, the encoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y;provide a set of inputs, at least one input of spatial size of XxY; for at least one input i in the set of inputs:(i) form AxB sub-blocks of size MxN from the input, for integers A, B, M, and N;(ii) determine at least two coefficients for at least one of the AxB sub-blocks based on the corresponding sub-block, wherein at least one of the at least two coefficients is based on at least two input values in the set of inputs; and(iii) re-arrange the at least two coefficients for at least one of the AxB sub-blocks to form a re-sized input of size of AxBxC(i), where C(i) < (M*N); and apply the neural network using the at least one re-sized input in the set of inputs to generate an output.

31. The encoder of claim 30, wherein the encoder is further configured to perform the method of any one of claims 2-19.

32. An encoder configured to apply filtering during video coding using a neural network, the encoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; provide a set of inputs, at least one input of spatial size of XxY; apply the neural network using the set of inputs to generate an output, wherein applying the neural network comprises, for a first layer having an input signal x and second layer having an output signal y in the one or more additional layers of the neural network:(i) form AxB sub-blocks of size MxN from the input signal x, for integers M and N;(ii) determine at least two coefficients for at least one of the AxB sub-blocks based on the corresponding sub-block, wherein at least one of the at least two coefficients is based on at least two input values from the input signal x; and(iii) re-arrange the at least two coefficients for the at least one AxB sub-blocks to form a re-sized input signal x of size of AxBxC(i), where C(i) < (M*N); and(iv) feed the re-sized input signal x into the first layer; and(v) re-arrange the output signal y of the second layer from a size of A'xB'xC'(i), where C'(i) < (M*N), to a single channel output.

33. The encoder of claim 32, wherein the encoder is further configured to perform the method of any one of claims 20-31.

34. A decoder configured to apply filtering during video coding using a neural network, the decoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; provide a set of inputs, at least one input of spatial size of XxY; for at least one input i in the set of inputs:(i) form AxB sub-blocks of size MxN from the input, for integers A, B, M, and N;(ii) determine at least two coefficients for at least one of the AxB sub-blocks based on the corresponding sub-block, wherein at least one of the at least two coefficients is based on at least two input values in the set of inputs; and(iii) re-arrange the at least two coefficients for at least one of the AxB sub-blocks to form a re-sized input of size of AxBxC(i), where C(i) < (M*N); and apply the neural network using the at least one re-sized input in the set of inputs to generate an output.

35. The decoder of claim 34, wherein the decoder is further configured to perform the method of any one of claims 2-19.

36. A decoder configured to apply filtering during video coding using a neural network, the decoder being further configured to: provide a neural network having a convolutional layer and one or more additional layers for filtering a block of image samples of spatial size XxY, for integers X and Y; provide a set of inputs, at least one input of spatial size of XxY; apply the neural network using the set of inputs to generate an output, wherein applying the neural network comprises, for a first layer having an input signal x and second layer having an output signal y in the one or more additional layers of the neural network:(i) form AxB sub-blocks of size MxN from the input signal x, for integers M and N;(ii) determine at least one coefficient for at least one of the AxB sub-blocks based on the corresponding sub-block, wherein the at least one coefficient is based on at least two input values from the input signal x; and(iii) re-arrange the at least one coefficient for the at least one AxB sub-blocks to form a re-sized input signal x of size of AxBx(M*N); and(iv) feed the re-sized input signal x into the first layer; and(v) re-arrange the output signal y of the second layer from a size of A'xB'xC'(i), where C'(i) < (M*N), to a single channel output.

37. The decoder of claim 36, wherein the decoder is further configured to perform the method of any one of claims 20-31.

Images