Early patch cropping for video encoding and decoding

Abstract

A method for processing an image. The method comprises selecting an input patch within the image. The input patch has a patch size (Yin, Xin) and Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction. The method further comprises processing the input patch using a neural network, NN, filter, thereby generating an output patch. The method further comprises modifying the image based at least on the output patch. The NN filter comprises convolutional layers C1-Cn arranged in a sequence. Processing the input patch using the NN filter comprises outputting from the convolutional layer Ck a first patch having a patch size of (Y1, X1). k is equal to or less than n, Y1 is a length of the first patch along the first direction and X1 is a length of the first patch along the second direction, Y1 = a1 x Yin, X1 = b1 x Xin, each of Y1, X1, Yin, and Xin is an integer that is greater than or equal to 1, and both a1 and 1 / a1 are non-integers and / or both b1 and 1 / b1 are non-integers.

Description

EARLY PATCH CROPPING FOR VIDEO ENCODING AND DECODINGTECHNICAL FIELD

[0001] This disclosure relates to early patch cropping for video encoding and decoding.BACKGROUND

[0002] Video Compression

[0003] Video is the dominant form of data traffic in today’s networks and is projected to continue to increase its share (see, e.g., reference [1]). One way to reduce the data traffic from video is compression. In the video compression, the source video is encoded to a bitstream, which then can be stored and transmitted to end users. Using a decoder, the end user can extract the video data from the bitstream and display it on a screen. However, since the encoder may not know what kind of device the encoded bitstream is going to be sent to, it has to compress the video to a predetermined format such as the standardized format Versatile Video Coding (VVC). This way, all devices that support the chosen standard can decode the video. Compression can be lossless, i.e., the decoded video will be identical to the source given to the encoder, or lossy, where a certain degradation of content is accepted. Using lossy compression allows for significantly lower bit rates, i.e., the compression ratio can be much higher. This is because reproducing image noise perfectly can make lossless compression quite expensive.

[0004] A video sequence contains a sequence of pictures. A color space commonly used in video sequences is YCbCr, where Y is the luma (brightness) component and Cb and Cr are the chroma components. Sometimes the Cb and Cr components are called U and V. Other color spaces are also used. These include ICtCp, IPT, constant-luminance YCbCr, RGB, YCoCg, etc. The embodiments of this disclosure are limited to a particular color space but can be applied to various different color spaces.

[0005] VVC and HEVC

[0006] VVC and its predecessor High Efficiency Video Coding (HEVC) are blockbased video codecs standardized and developed jointly by The International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and Moving Picture Experts Group (MPEG). The codecs utilize both temporal and spatial prediction. VVC and HEVC are similar in many aspects. Spatial prediction is achieved using intra (I)prediction from within the current picture. Temporal prediction is achieved using unidirectional (P) or bi-directional inter (B) prediction on the block level from previously decoded reference pictures. In the encoder, the difference between the original pixel data and the predicted pixel data, referred to as the residual, is transformed into the frequency domain, quantized and then entropy coded before transmitted together with necessary prediction parameters such as prediction mode and motion vectors, also entropy coded. The decoder performs entropy decoding, inverse quantization and inverse transformation to obtain the residual, and then adds the residual to the intra or inter prediction to reconstruct a picture. The VVC version 1 specification was published as Rec. ITU-T H.266 | ISO / IEC 23090-3, “Versatile Video Coding,” in 2020.

[0007] Blocks and Units

[0008] In many video coding standards, such as HEVC and VVC, each component is split into blocks and the coded video bitstream consists of a series of coded blocks. A block is a two-dimensional array of samples. It is common in video coding that the picture is split into units that cover a specific area of the picture. Each unit consists of all blocks from all components that make up that specific area and each block belongs fully to one unit. The macroblock in H.264 and the Coding unit (CU) in HEVC and VVC are examples of units.

[0009] A block can alternatively be defined as a two-dimensional array that a transform used in coding is applied to. These blocks are known under the name “transform blocks.” Alternatively, a block can be defined as a two-dimensional array that a single prediction mode is applied to. These blocks can be called “prediction blocks.” In this disclosure, the word “block” is not tied to one of these definitions but that the descriptions herein can apply to either definition.

[0010] Exploration Experiment on Neural Network based Video Coding (NNVC)

[0011] At the 20th JVET (Joint Video Experts Team) meeting it was decided to set up an exploration experiment (EE) on neural network-based (NN-based) video coding (see, e.g., reference [5]). The exploration experiment continued at the subsequent JVET meetings 21 through 31 with many tests: NN-based in-loop filtering, NN-based post filtering, NN-based super resolution and NN-based intra prediction. The first two are of special interest to the disclosure below and will be described further.

[0012] VVC contains three in-loop filters that are not based on neural networks: Adeblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter (ALF). The deblocking filter is used to remove block artifacts by smoothening discontinuities in horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS can have values 0, 1, and 2, where a larger value indicates a stronger filtering. The output of the deblocking filter is further processed by the SAO filter, and the output of the SAO filter is then processed by the ALF. The output of the ALF can then be put into the decoded picture buffer (DPB), which contains decoded pictures that may be used for prediction of subsequently encoded (or decoded) pictures. Since the deblocking filter, SAO filter and ALF filter in this way influence the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loop filters. This means that changes done by the loop filter may influence not only the current picture but future pictures. It is possible for a decoder to further filter the picture in the DPB, but not store the filtered output in the DPB. In contrast to loop filters, such a filter is not influencing future predictions and is therefore classified as a post-processing filter, also known as a postfilter. Postfiltering is generally optional for decoders and thereby not required to be performed for decoder implementations to conform to a standard specification.

[0013] The contributions JVET-X0066 and JVET-Y0143 (references [2] and [3], respectively) are two successive contributions that describe NN-based in-loop filtering. Both contributions use the same NN models for filtering. The NN-based in-loop filter is placed before the SAO filter and the ALF and the samples before the deblocking filter are used as input to the filter. The output of the NN-based in-loop filter is mixed with the output of the deblocking filter and forwarded as the input to the SAO filter. The purpose of using the NN- based filter is to improve the quality of the reconstructed samples. Here it is helpful that the NN model is non-linear. While deblocking, the SAO filter and the ALF all contain nonlinear elements such as conditions, and thus are not strictly linear, meaning that all three of them are based on linear filters. In contrast, a sufficiently large NN model can in principle learn any non-linear mapping and is therefore capable of representing a wider class of functions compared to the deblocking filter, the SAO filter, and the ALF. In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters. In a refined version of that work presented in the contribution JVET-AB0052, only two models are used: One for luma samples and another model for chroma samples.

[0014] JVET-AD0380 (reference [4]) proposed a new unified design for the NN-basedin-loop filtering, which captures the benefits of previous NN structures. The unified filter has only one NN model, to filter luma and chroma samples and intra and inter pictures. FIG. 12 shows the structure of the unified filter. In the unified filter, there are six inputs - the reconstructed samples of luma and chroma before deblocking (‘rec’), the prediction samples of luma and chroma (‘pred’), the BS information of luma and chroma (‘bs’), the quantization parameter for a sequence (‘QPbase’), the quantization parameter for each slice ('QPslice’) as well as information on whether a particular sample was intra-predicted, uni-predicted or bipredicted (‘IPB’). These inputs first go through a convolutional layer (3x3 or 1x1) and a parametric rectified linear unit (PReLU) layer separately, and then they are concatenated and fused together with a 1x1 convolutional layer.

[0015] In FIG. 12, a convolutional layer with a kernel size of 3x3 is denoted “conv 3x3” and a convolutional layer with a kernel size of 1x1 is denoted “conv 1x1.” The values dl, d2, ... are the numbers of the output channels. “J,2” denotes downsampling with a factor of 2. There are N backbone (a.k.a., “back bone”) blocks. Inside a backbone block, a convolutional layer “conv 3x1 CxC21” means that the convolution kernel is of size 3x1, the number of input channels is C, and the number of output channels is C21.

[0016] The NN-based in-loop filters presented in JVET-X0066, JVET-AB0053, JVET- AB0052, and JVET-AD0380 increase the compression efficiency of the codec substantially, i.e., they lower the bit rate substantially without lowering the objective quality as measured by MSE-based Peak Signal-to-Noise Ratio (PSNR). Increases in compression efficiency, often referred to simply as “gain,” are often measured as the Bjontegaard-delta rate (BDR) against an anchor. As an example, a BDR of -1% means that the same PSNR distortion can be reached with 1% bitrate saving on average. As reported in JVET-AF0041, for the random access (RA) configuration, the BDR for the luma component (Y) is -10.27%, and for the all- intra (Al) configuration, the BDR for the luma component is -7.86%. The complexity of NN models used for compression are often measured by MAC / pixel (multiply-accumulate operations per pixel). The high bitrate savings of an NN model is typically directly related to the high complexity of the NN model. The model described in JVET-AF0041 has a complexity of 477 kMAC / pixel, i.e., 477,000 multiply-accumulate operations per pixel. There are also other measures of complexity, such as total model size in terms of stored parameters.

[0017] A Low Operation Point (LOP) architecture NN loop filter currently used in aJVET EE test (EE1-E0) is illustrated in FIG. 13 (from JVET-AG2023). In FIG. 13, the input patch size is 144x144 and there is a final cropping step that crops 8 pixels from each side of the output luma patch and 4 pixels from each side of each chroma patch (which has half the size of the luma patch). Hence the final patch size after cropping in the output is equal to 128x128 for luma and 64x64 for chroma branch. The complexity of this current model is about 17 kMULT per sample. This means that about 17,000 multiplications are performed for calculating one sample value in the output of neural network loop filter architecture in FIG. 13.

[0018] While FIG. 13 shows the way that this cropping from 144x144 to 128x128 is performed on a conceptual level, the way it is implemented in the reference software NNVC version 7 is slightly different: FIG. 14 shows a full picture 1401 (for instance, of size 1920x1080 samples) of luma samples before the deblocking filter. A 144x144 patch of samples 1402 is taken from this picture. This patch contains the 128x128 patch of samples 1403 that are to be influenced using the filtering.

[0019] As shown in FIG. 15, the 144x144 patch of samples 1402 is then input into the neural network filter as the input RBCEXTY 1501. Chroma values 1502 are also input, as are Pred, BS, QPbase, QPSlice, and IPB values. The neural network processes these inputs and produces a luma output 1503 of patch size 144x144 and a chroma output 1504 of patch size 72x72. Here, for simple explanation, the explanations below will be focused on the luma output residual 1503. This luma output residual has patch size 144x144, but only the middle 128x128 residual samples are used, and they are added to the 128x128 samples 1403 in picture 1401 in FIG. 14. Therefore, this adding operation is only performed 128x128 times. This is in contrast to FIG. 12, which shows this adding operation operating on patches of size 144x144 samples (with the output of pixel shuffle, which is of patch size 144x144, as the first input, and the input RBCEXTY, which also has patch size equal to 144x144, as the second input) and only cropping to 128x128 afterwards. The result is identical, but in the way the code is written, which is in accordance with FIGS. 14 and 15, only 128x128=16384 additions are needed, whereas in FIG. 13, 144x144 = 20736 additions are needed.

[0020] In JVET standardization, development and study of codecs are done using a common test conditions (CTC). The CTC specifies how a codec under test should be configured and what test sequences that should be used. Keeping the test conditions static enables apples-to-apples evaluations, but with the drawback that configurations outside theCTC are not tested much and the codec performance may become too optimized towards the CTC.

[0021] Input and Output Patch Sizes in the State-of-the-Art for NN-based In-loop Filters

[0022] In block-based image and video codecs, the input picture is usually divided into blocks and each block is processed separately through the pipeline which makes the codec easily extendable to larger resolutions. To provide a more seamless transition in the borders of the neighboring blocks, and to help the NN filter improve samples near the edge better, the NN filters are applied on a larger patch of pixels in a picture and then the output of the filter is cropped to the proper size. With this approach the information from the pixels in the neighboring blocks which are close enough to the border will also be considered in the filtering process of the current block. This is helpful for when filtering a sample next to the edge, e.g., the top-left sample of the block, since this sample will now have samples to the left of it and above it, and the filter can use these samples to help in the filtering. If no such extra margin was used, the NN filter would not know how the image continues to the left and above, and would have had more difficulty in improving such a sample.

[0023] As an example, when filtering a block of samples of the size 128x128, eight samples on either side of the filter may be used in the input so that the input is (128+8+8)x(128+8+8) = 144x144, a margin of 8 samples everywhere. Choosing a good size of the margin is a trade-off: With a small margin or no margin, the quality of the filtering suffers. On the other hand, with a large margin, complexity increases.

[0024] In the current state-of the art for NN-based in-loop filters for video compression / decompression, the size of the output is kept the same as the input. As an example, if the input can be 144x144 samples big, the output is also 144x144 samples big. As a final step after the filtering, the middle 128x128 samples are cropped from the output and used by the encoder / decoder. To keep the size, if a 3x3 kernel is used, one sample on each side of the input is padded (usually with zeros) so that the output size will be the same as the input size for that layer. In PyTorch, this is done by setting the input padding=(l,l) when invoking the conv2d function. If a 5x5 kernel is used, padding=(2,2) must be used to keep the input and output sizes of the layer the same. Likewise, if a 3x1 kernel is used, padding=(l,O) will keep the input and output sizes of the layer the same.

[0025] FIGS. 16A and 16B show that using padding=(l,l) will pad with zeros 1samples outside the block everywhere. This will keep the same dimensions after a 3x3 convolution.

[0026] In the current state-of the art for NN-based in-loop filters for video compression / decompression, even though the size of the output is kept the same as the input, the sizes of the inputs inside the network sometime change by a factor of two. As an example, in FIG. 12, one of the layers is denoted - CON\%2 3x3.

[0027] This layer uses stride=(2,2) (as indicated by 2) in order to reduce the resolution of the output by a factor 2. If the input to this layer is 144x144, the output of the layer is 72x72. This layer can have any number of channels as input and output, but it is common to increase the number of channels when reducing the spatial resolution. For instance, the LOP2 model goes from 144x144x24 (24 channels) to 72x72x32 (32 channels).

[0028] Also in FIG. 12, one of the layers is denoted - Pixel Shuffle. This layer increases the resolution by a factor of two. For example, if the input to this layer is 72x72x4, the output of the layer will be 144x144x1, which rearranges the samples from four channels of spatial size 72x72 into one channel of spatial size 144x144. When using pixelshuffle it is not possible to decide the number of output channels. If the pixel shuffle is changing the resolution by a factor of four (two in width and two in height as in this case), the number of output channels will also have to decrease by a factor of four.SUMMARY

[0029] Certain challenges presently exist. For example, high complexity is a major issue with current models of the NN loop filters or the integrated NN filters (hereinafter just “NN filters”). It is a major barrier against having them implemented and integrated into everyday devices. Thus, kMAC / kMULT as measures of complexity are desired to be kept down. As mentioned before, the NN filters are currently applied on a larger patch of pixels in a picture and then the output of the filter is cropped to the proper size. Having a larger patch size is equivalent to having large margins, which increase the kMAC / kMULT considerably since the complexity is often proportional to the number of samples in the input. For instance a margin of 8 for a 128 xl28 patch size would increase the complexity by (128+8+8)x(128+8+8) / (128xl28) = 1.26, i.e., an increase by 26% compared to no margin, and doubling the margin to 16 would likewise give an increase of 56%.

[0030] Another problem with existing technology is high energy consumption which is directly related to the complexity of the network of the NN filter. The high energyconsumption is an important factor in both training and inference stages.

[0031] Thus, in using the NN filters to process pictures for video encoding and decoding, it is important to find an appropriate balance between the performance of the NN filters and the complexity / the high energy consumption of the NN filters. Note that the complexity of processing a picture is usually quantified by the number of multiply- accumulate operations per pixel kMAC / pix (or kMAC for short) or the number of multiplications per pixel kMULT / px (or kMULT for short), and the performance for a codec is defined by increases in compression efficiency, often referred to simply as “gain” and is often measured as the Bjontegaard-delta rate (BDR) against an anchor. As an example, a BDR of -1% means that the same PSNR distortion can be reached with 1% fewer bits.

[0032] In order to solve the above problem, in the embodiments of this disclosure, there are provided different ways of lowering the complexity and energy consumption of an NN filter used for processing a picture while not sacrificing the performance substantially.

[0033] More specifically, in one aspect of the embodiments of this disclosure, there is provided a method for processing an image. The method comprises selecting an input patch within the image. The input patch has a patch size (Yin, Xin) and Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction. The method further comprises processing the input patch using a neural network, NN, filter, thereby generating an output patch. The method further comprises modifying the image based at least on the output patch. The NN filter comprises convolutional layers Ci-Cn arranged in a sequence. Processing the input patch using the NN filter comprises outputting from the convolutional layer Ck a first patch having a patch size of (Yl, XI). k is equal to or less than n, Yl is a length of the first patch along the first direction and XI is a length of the first patch along the second direction, Yl = al x Yin, XI = bl x Xin, each of Yl, XI, Yin, and Xin is an integer that is greater than or equal to 1, and both al and 1 / al are non-integers and / or both bl and 1 / bl are non-integers.

[0034] In another aspect, there is provided a method for processing an image. The method comprises selecting an input patch within the image, wherein the input patch has a patch size, processing the input patch using a neural network, NN, filter, thereby generating an output patch, and modifying the image based at least on the output patch. The NN filter comprises processing blocks arranged in a sequence, the processing blocks comprise a processing block R, and the processing block R comprises layers LI -Ln arranged in asequence. Processing the input patch using the NN filter comprises: inputting a first patch to the processing block R, wherein the inputted first patch is provided to the layer LI of the processing block R; cropping the inputted first patch, thereby generating a cropped first patch and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch. A size of the cropped first patch is smaller than a size of the first patch.

[0035] In a different aspect, there is provided a computer program comprising instructions which when executed by processing circuitry cause the processing circuitry to perform the method of any one of the above embodiments.

[0036] In a different aspect, there is provided a carrier containing the computer program of the above embodiment. The carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0037] In a different aspect, there is provided an apparatus for processing an image. The apparatus is configured to select an input patch within the image. The input patch has a patch size (Yin, Xin) and Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction. The apparatus is further configured to process the input patch using a neural network, NN, filter, thereby generating an output patch; and modify the image based at least on the output patch. The NN filter comprises convolutional layers Ci-Cn arranged in a sequence. Processing the input patch using the NN filter comprises outputting from the convolutional layer Ck a first patch having a patch size of (Yl, XI). k is equal to or less than n, Y1 is a length of the first patch along the first direction and XI is a length of the first patch along the second direction, Yl = al x Yin, XI = bl x Xin, each of Yl, XI, Yin, and Xin is an integer that is greater than or equal to 1, and both al and 1 / al are non-integers and / or both bl and 1 / bl are non-integers.

[0038] In a different aspect, there is provided an apparatus for processing an image. The apparatus is configured to select an input patch within the image, wherein the input patch has a patch size, process the input patch using a neural network, NN, filter, thereby generating an output patch, and modify the image based at least on the output patch. The NN filter comprises processing blocks arranged in a sequence, the processing blocks comprise a processing block R, and the processing block R comprises layers LI -Ln arranged in a sequence. Processing the input patch using the NN filter comprises: inputting a first patch to the processing block R, wherein the inputted first patch is provided to the layer LI of theprocessing block R; cropping the inputted first patch, thereby generating a cropped first patch and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch. A size of the cropped first patch is smaller than a size of the first patch.

[0039] In a different aspect, there is provided an apparatus comprising processing circuitry and a memory, said memory containing instructions executable by said processing circuitry, whereby the apparatus is operative to perform the method of any one of the above embodiments.

[0040] The embodiments of this disclosure allow lowering the computational complexity of the NN model in the NN filter, e.g., providing 0.5% complexity reduction in the Low Operation Point (LOP) architecture NN loop filter, while keeping the performance of the NN filter the same or at the acceptable level.

[0041] The embodiments also allow speeding up the inference step of the NN model by avoiding unnecessary calculations, thereby reducing the total number of calculations performed by the NN model. In case the embodiments are used for the initial NN model before training, the embodiments may also speed up the training time as compared to the existing NN model. This improvement comes from a lower number of required calculations that are achieved by the embodiments. By avoiding the unnecessary calculations, the energy usage of the NN model can be lowered during the training stage. Note that the embodiments below can be applied directly at the inference stage, after training with the current state of the art model architecture is completed, without affecting the training and so the previous trainings can be reused.

[0042] The embodiments below are applicable to most of the current NN-based inloop filtering architectures.BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0044] FIG. 1A shows a system according to some embodiments.

[0045] FIGS. IB and 1C show examples of the system shown in FIG. 1A.

[0046] FIGS. 2A and 2B show how the size of a patch inputted to a layer can be reduced.

[0047] FIGS. 3 A and 3B show effects of the size of an input patch on an output patch.

[0048] FIG. 4 shows an example of an LOP2 neural network.

[0049] FIG. 5 shows a structure of an NN loop filter according to some embodiments.

[0050] FIGS. 6A-6B illustrate “valid padding” (a.k.a., “no padding”).

[0051] FIGS. 6C-6D illustrate how some computation can be saved by filtering samples on a border.

[0052] FIG. 7 shows a structure of a back bone (a.k.a., “backbone”) block.

[0053] FIG. 8A shows a structure of a back bone block.

[0054] FIG. 8B illustrates how a cropping operation can be performed for reconstruction.

[0055] FIGS. 9A and 9B illustrate how a dummy convolutional layer can be used to perform a cropping operation.

[0056] FIG. 10A shows a process according to some embodiments.

[0057] FIG. 10B shows a process according to some embodiments.

[0058] FIG. 11 shows an apparatus according to some embodiments.

[0059] FIG. 12 shows a unified neural network (NN) filter.

[0060] FIG. 13 shows an LOP architecture NN loop filter.

[0061] FIG. 14 illustrates how a portion of a picture is processed using a patch.

[0062] FIG. 15 shows an example NN loop filter.

[0063] FIGS. 16A and 16B show the concept of padding.

[0064] FIG. 17A shows a simplified view of an exemplary structure of an NN filter according to some embodiments.

[0065] FIG. 17B shows a separate cropping layer according to some embodiments.

[0066] FIGS. 18A-18B show the concept of applying a convolution operation on a 3x3 input patch using a 2x2 kernel.

[0067] FIGS. 19A-19B show the concept of applying a convolution operation on a 5x5input patch using a 4x4 kernel without padding.

[0068] FIGS. 20A-20B and 21 show the concept of applying a convolution operation on a 5x5 input patch using a 4x4 kernel with (1,1) padding.

[0069] FIG. 22 shows the concept of applying a cropping operation in a feed forward path.DETAILED DESCRIPTION

[0070] In this disclosure, the terms “neural network,” “neural network model,” and “NN model” are used interchangeably. Also, in this disclosure, “patch size” is used as a general term for referring to the size of the data patch in different arrangements, e.g., with various number of channels. There might be layers or functionalities in the neural network that rearrange the dimensions of data, such as pixel shuffling, and so speaking about reducing the patch size also includes different arrangements of data before and after those layers or functionalities.

[0071] For example, let’s assume that there is a first input having a width and a height of 144, and 8 channels. The first input can be denoted as 144x144x8. Let’s further assume that there is a second input having the width and height of 144 but 4 channels. The second input may be denoted as 144x144x4. Then the first and second inputs have the same patch size but different number of channels.

[0072] It should be noted that the size 144x144 is just an example, which is used when processing an area of 128x128 samples with a margin of 8. It is possible to use the same neural network with different sizes. If we want to process an area of 256x256 samples, we would input (256+8)x(256+8)=272x272 samples to the neural network. All patch sizes and kernel sizes mentioned below are provided only for simple explanation purpose and do not limit the embodiments of this disclosure in any way.

[0073] FIG. 1A shows a system 100 according to some embodiments. The system 100 comprises a first entity 102, a second entity 104, and a network 110. The first entity 102 is configured to transmit to the second entity 104 a video stream (a.k.a., “a video bitstream,” “a bitstream,” “an encoded video”) 106.

[0074] The first entity 102 may be any computing device (e.g., a network node such as a server) capable of encoding a video using an encoder 112 and transmitting the encoded video to the second entity 104 via the network 110. The second entity 104 may be any computing device (e.g., a network node, a user equipment such as a mobile phone, a laptop,a tablet, a vehicle, etc.) capable of receiving the encoded video and decoding the encoded video using a decoder 114. Each of first entity 102 and second entity 104 may be a single physical entity or a combination of multiple physical entities. The multiple physical entities may be located at the same location or may be distributed in a cloud.

[0075] In some embodiments, as shown in FIG. IB, the first entity 102 is a video streaming server 132 and the second entity 104 is a video streaming client — e.g., a user equipment (UE) 134. The UE 134 may be any one of a desktop, a laptop, a tablet, a mobile phone, a head-mounted display or any other computing device. The video streaming server 132 is capable of transmitting a video bitstream 136 (e.g., a movie) to the video streaming client 134. Upon receiving the video bitstream 136, the video streaming client 134 may decode the received video bitstream 136, thereby generating and displaying a video for the video streaming.

[0076] In other embodiments, as shown in FIG. 1C, the first entity 102 and the second entity 104 are first and second UEs 152 and 154. In one example, the first UE 152 is an offeror of a video conferencing session or a caller of a video chat, and the second UE 154 is an answerer of the video conference session or the answerer of the video chat. In the embodiments shown in FIG. 1C, the first UE 152 is capable of transmitting a video bitstream 156 for a video conference or a video chat to the second UE 154. Upon receiving the video bitstream 156, the second UE 154 may decode the received video bitstream 156, thereby generating and displaying a video for the video conferencing session or the video chat.

[0077] As explained above, in encoding and decoding a video, an NN filter may be used. However, due to the complexity of operations performed by networks in the NN filter, encoding and decoding a video using the NN filter may not be efficient. Therefore, there is a need for a way to reduce the operational complexity of the NN filter while keeping the performance of the NN filter at the acceptable level.

[0078] Accordingly, in one aspect of the embodiments of this disclosure, the size of a patch inputted to each of one or more layers of the NN filter is reduced. By reducing the size of a patch inputted to each of one or more of the layers, the operational complexity of the NN filter can be reduced. FIGS. 2A and 2B show how the size of a patch inputted to a layer can be reduced.

[0079] In the NN filter shown in FIG. 2A, the dimensions (size) of a patch inputted tothe NN filter are same as the dimensions of a patch outputted from the NN filter when the number of channels is disregarded - i.e., Co may be different from Cn. Note that, in the NN filter shown in FIG. 2A, any change in the size of a patch is a factor of two. For example, in FIG. 2A, the layer 2 changes the size of a patch from 144x144 to 72x72 and the layer n-2 changes the size of a patch from 72x72 back to 144x144. This is in essence a change in resolution. In this disclosure, a size change of a patch is called a resolution change if the change in patch size is an integer. As an example, since the layer 2 changes the patch size from 144x144 (input) to 72x72 (output) and 144 / 72 = 2 is an integer, the layer 2 is considered to perform a resolution change. Similarly, since the layer n-2 changes the patch size from 72x72 (input) to 144x144 (output) and 144 / 72 = 2 is an integer, the layer n-2 is considered to perform a resolution change

[0080] As mentioned above, in the NN filter shown in FIG. 2A, the size of the input patch and the size of the output patch are the same. On the contrary, in the NN filter shown in FIG. 2B, the size of the input patch (i.e., 144x144) is different from the size the output patch (i.e., 136x136). This difference in patch size is achieved by the layers included in the NN filter. For example, the layer n-1 in FIG. 2B changes the patch size by a factor that is not an integer - i.e., from 140x140 to 138x138. Note that here, neither 144 / 138=1.04 nor 138 / 144 = 0.958 is an integer. In another example, the layer n in FIG. 2B changes the patch size by a factor that is not an integer - i.e., from 138x138 to 136x136.

[0081] Reducing the patch size in the NN filter by a non-integer factor is to create a better trade-off between the operational complexity of the NN filter and the performance of the NN filter - e.g., the quality of image processing performed by the NN filter.

[0082] Note that the patch size reduction according to the embodiments of this disclosure is different from simply rearranging samples or changing the resolution by a factor of an integer. One of the goals of the patch size reduction according to the embodiments of this disclosure is to remove sample values that do not affect the useful parts of the output of the NN filter before those sample values are processed by layers of the NN filter, thereby preventing the sample values from increasing the operational complexity of the NN filter.

[0083] Another difference between the existing NN filter and the NN filter according to the embodiments of this disclosure is how a change in the size of a patch inputted to a convolutional layer affects the size of a patch outputted from the convolutional layer. For example, in the existing NN loop filter, a difference between the size of an input patch andthe size of an output patch varies proportionally in accordance with a change in the size of the input patch. In contrast, in the embodiments of this disclosure, the patch size is trimmed by a fixed number regardless of the size of the input patch, thereby resulting in that the difference between the size of the input patch and the size of the output patch is maintained regardless of the size of the input patch. In other words, In the existing art, the input size (x) and output size (y) have a linear relationship of y = a * x. Thus, in the existing art, if the input size is 144, the output size is 144 and if the input size is reduced to 128, then the output size will be reduced to 128. The output size is a constant factor of the input size, here a factor of 1, as 144 / 144 = 128 / 128 = 1. On the contrary, in some embodiments of this disclosure, the input size and output size have a non-linear relationship that does not follow y = a * x + b. Thus, a change from 144 to 128 in the input size will result in the output size change from 136 to 120, where the input size and output size are not a constant factor of each other: 144 / 136 is different from 128 / 120. This is illustrated in FIGS. 3A and 3B.

[0084] In FIG. 3B, when an input patch having the size of 144 x 144 is processed through layers of the NN filter, the NN filter outputs an output patch having a reduced size of 136 x 136. The reduced size of 136 x 136 is a reduction of 8 in both X and Y dimensions from the input size of 144 x 144. Similarly, when an input patch having the size of 128 x 128 is processed through layers of the NN loop filter, the NN loop filter outputs an output patch having a reduced size of 120 x 120. Like the previous example, the reduced size of 120 x 120 is a reduction of 8 in both X and Y dimensions from the input size of 128 x 128.

[0085] Concept of Performing Patch Size Reduction in the Tail part

[0086] Another concept of the embodiments of this disclosure is performing the patch size reduction in the tail part of the NN loop filter. This is illustrated in FIG. 4.

[0087] FIG. 4 shows an LOP2 neural network as implemented in the NNVCv7 code. In the existing art, the input 401 of the NN filter has the patch size of 144x144, the output 405 of the last back bone layer of the NN filter has the patch size of 72x72, and each of the outputs 406, 407, 408, 409, and 410 of the subsequent layers CONV 1x3, CONV 3x1, CONV 1x1, PReLU, and CONV 3x3 also has the patch size of 72x72. Finally, the Pixel Shuffle layer which receives the output 410 of the patch size of 72x72 changes the resolution so that the output 403 of the NN filter has the patch size of 144x144. The middle portion having the size of 128x128 within these 144x144 samples are then added to thesamples 1403 of FIG. 14 to get the final output.

[0088] In contrast, according to some embodiments of this disclosure, the output 405 of the last back bone block has the patch size of 72x72, but the CONV 1x3 layer reduces the patch size to 72x70 - i.e., the output 406 has the patch size of 72x70 -, and CONV 3x1 layer reduces the patch size to 70 x 70 - i.e., the output 407 has the patch size of 70x70. The subsequent two layers - CONV 1x1 and PreLU - do not change the patch size, and thus each of the outputs 408 and 409 has the patch size of 70 x 70. The last convolutional layer CONV 3x3 further reduces the patch size to 68x68 - i.e., the output 410 has the patch size of 68x68. Finally, the Pixel Shuffle layer changes the resolution of the patch by a factor of two from 68x68 to 136x136. Thus, overall, the patch size is changed by a noninteger factor.

[0089] In these embodiments, the patch size is changed several times before the last convolution layer. This means that the last convolution layer ‘CONV 3x3’ will only need to perform 3X3X68X68XCYX4 multiplications. On the contrary, in the existing art, that same layer will need to perform 3X3X72X72XCYX4 multiplications, which is 12% more.Likewise, similar savings are possible for earlier layers. For instance, the CONV 1x1 layer in the above example operates on the patch size of 70x70 instead of on the patch size of 72x72 as in the existing art. This gives a change of 70*70 / (72*72)-! = -5% in the number of multiplications performed in that layer since the patch size has been changed before it.

[0090] This patch reduction aims at a better complexity-performance trade-off in the NN filter. More specifically, according to some embodiments of this disclosure, the patch reduction is performed by pruning the excess calculations on the samples not contributing to the sample values in the useful portion of the output of the NN filter, thereby directing the calculations only to the sample data that is actually used in the output of the neural network. The resulting complexity reduction budget may be used in improving the performance of the NN filter, for example, by increasing the number of the parameters or layers in the NN filter.

[0091] General Solution

[0092] In order to reduce the operational complexity of the NN filter, in some embodiments of this disclosure, the NN model of the NN filter is modified to remove excess calculations such as calculations performed on the border samples which are later cropped. The excess calculations here are defined as the calculations that are present in the inference but do not substantially affect the useful parts of the output of the NN filter, forexample, sample values inside the cropping area that are later going to be used.

[0093] By removing samples from the borders of a patch, the patch size may be reduced. In the embodiments of this disclosure, the patch size reduction is fixed and not proportional to the patch size itself. This means that even when the same neural network gets a different size patch as the input patch, the amount of the patch size reduction is not affected.

[0094] For example, let’s assume that the input patch size to layer n is 144x144 and the input patch size to layer n+1 after layer n is 142x142 which is 2 sample points smaller in X and Y dimensions. In this example if the input patch size to layer n becomes 72x72, the input patch size to layer n+1 after layer n becomes 70x70 which is still 2 sample points smaller in X and Y dimensions.

[0095] More generally, let’s assume that there are at least two layers LI and L2 in the neural network where layer L2 is after layer LI in the neural network, the output patch size of layer L2 is smaller than the output patch size of layer LI, and the output patch size of layer L2 is not a divisor of the output patch size of layer LI. For example, see the patch size change between the output of layer n-2 and the output of layer n-1 in FIG. 2B. This is a size change that is not simply coming from rearrangement of the samples, such as a pixel shuffle / pixel unshuffle operation. Also, it is not a size change resulting from having a stride parameter being different from 1. The patch size reduction in these embodiments is not proportional to the patch size but fixed.

[0096] Note that a patch size adjustment can be implemented with cropping or other mechanisms. However, in this disclosure, the term “cropping” is used as an umbrella term for different kinds of size adjustment.

[0097] In some embodiments, a cropping step, e.g., cropping of several border pixels right before outputting, is replaced by cropping the excess calculations earlier in the previous layers of the neural network. One example is to replace a final cropping step that crops a (n+2) x (n+2) patch size into an n x n patch size with a cropping step that happens right before the final 3 x 3 convolution layer and crops the size of the input patch to the last 3 x 3 convolution into (n+2) x (n+2). By cropping before the last convolution step, the last convolution is applied on a smaller patch size which results in fewer calculations while providing practically the same output as before. In the above example it is assumed that the padding is not applied to the final 3x3 convolution (or a zero padding is applied) and so the output patch size is not affected by padding but only by the input patch size and the size ofthe kernel.

[0098] Alternatively or additionally, a cropping step can be added to the feedforward path, bringing the input patch forward to be added to the output patch. The added cropping step crops the input patch to the size equal to the output patch size before addition step. There may be crop steps added for each feed forward path, e.g., one for luma values and another for chroma values.

[0099] For example, the LOP architecture of FIG. 13 may be replaced by the architecture in FIG. 5, where the final crop boxes for both luma and chroma branches are moved before the final convolution step, hence reducing the number of calculations in the last convolution step in both luma and chroma channels. Also, two crop boxes are added to the feedforward paths for luma and chroma to match the patch sizes before summation of the values. This new architecture results in fewer calculations but the same final output, hence the same quality of the output.

[0100] Instead of or in addition to providing separate cropping layer(s), the cropping effect can also be achieved by adjusting the padding mechanism, e.g., by changing the number of padded pixels. More specifically, a convolutional layer with a k x k kernel size and zero padding will result in patch size reduction of [k / 2] pixels from each side. For example, when k = 5 with no padding, the size of the output patch from the convolutional layer will be [5 / 2] = 2 pixels smaller on each side. In this example, using a padding of 2 pixels from each side will result in the output patch size of the convolutional layer being equal to the input patch size to the convolutional layer.

[0101] In some embodiments, a cropping step, e.g., cropping of several border pixels right before outputting, is replaced by several smaller cropping steps in previous layers.Each of these smaller cropping steps may be a real cropping step introduced to the signal or be the result of a convolution layer without padding or with smaller padding, e.g., one pixel from each side of the data patch is cropped in each 3 x 3 convolution step when there is no padding. One method to implement this gradual cropping is to count back the required size of the data patch from the last layer. In one example, if the desired output size is a patch of n x n pixels, and the current implementation provides a (n+8) x (n+8) calculated data patch which is then cropped to an n x n size patch, then the input to the last 3 x 3 convolution layer requires no more than (n+2) x (n+2) pixels, and the input to the 3 x 3 convolution layer before the last convolutional layer requires no more than (n+4) x (n+4) pixels, and so on. In another example, if the input patch size to the neural network has the size of (n+16)x (n+16), then a final cropping step of 16 pixels (8 from each side) could be achieved by simply not padding the last eight convolution layers each having a 3 x 3 kernel size. Applying the proposed solution, the size of the output patch will be n x n directly. As a result, the excess calculations of those border pixels are avoided without affecting the important part of the neural network output. Note that, in PyTorch, no padding can be implemented by setting padding=(0,0).

[0102] In some embodiments, the cropping implementation of the luma (Y) branch and the cropping of the chroma (UV) branch can be the same or different. The final output size of luma and chroma can be the same but cropping may be performed differently for the luma branch and the chroma branch. In one example, the luma branch is cropped in the last two layers with a certain size while the chroma branch is cropped in the last three layers or in last two layers but with a different size.

[0103] Cropping By Padding

[0104] As briefly mentioned above, one way to perform a cropping operation, thereby avoiding unnecessary computations is by choosing a padding different than the “same” padding. In this disclosure, this is referred to as “cropping by padding.” The “same” padding is a padding when the input patch size to a convolutional layer and the output patch size of that convolutional layer are the same.

[0105] As an example, let’s assume that the last convolutional layer in the NN filter takes an input of size 144x144 samples and uses a convolutional layer with a 3x3 kernel with padding=(l,l) to produce an output of 144x144 samples. Finally, this is cropped by extracting the middle 128x128 samples. However, using padding=(l,l) can be wasteful since the top-most, bottom-most, left-most and right-most samples from the output will never be used due to the subsequent cropping to 128x128. Thus, according to some embodiments of this disclosure, padding=(0,0), i.e., no padding in the last layer is used instead. This will produce a smaller output of 142x142 samples, but since only the middle 128x128 samples are used due to the subsequent crop, the result after cropping is the same. A padding of (0,0) is sometimes referred to as ‘valid’ padding, since it only outputs values in positions that are valid in the sense that all parts of the filter are inside the input.Another term for ’valid’ padding is ‘no padding’ since the input is not extended before filtering.

[0106] FIGS. 6A and 6B show how padding=(0,0) works. If the input has the size of nxn and the kernel size is 3x3, the output will have the size of size (n-2)x(n-2). This willsave computation, since the number of multiplications needed for padding=(l,l) will be 3x3xl44xl44xCinxCout (here Cin is the number of input channels to the last layer, and the Cout is the number of output channels from the last layer), whereas with padding=(0,0) the number of multiplications is 3x3xl42xl42xCinxCout, which is a reduction of 2.75%. It should be noted that the number of multiplications needed to produce the output when padding=(l,l) can be lowered somewhat in the case when the block is padded by zeros compared to the figure of 3x3xl44xl44xCinxCout reported here. This is due to the fact that some multiplications can be saved on border pixels as shown in FIGS. 6C and 6D. As shown in FIGS. 6C and 6D, for three out of the nine positions in the 3x3 filter, the result of the convolution will be zero since the filter coefficients are going to be multiplied by zeros. So, in this case, instead of requiring 3x3 = 9 multiplications, only 2x3=6 multiplications are needed. However, this is still a lot more than avoiding calculations altogether (as compared to multiplying the coefficients with zeros). To simplify the notation, such savings will be ignored in the rest of the document by removing the 3x3 term on both sides, saying instead that the complexity goes down from 144xl44xCinxCout to 142xl42xCinxCout.

[0107] It also follows that if a final output having the size of 128x128 is desired, and if the last layer is a 3x3 kernel, it is wasteful to input anything bigger than 130x130 to the last layer. By avoiding padding in a strategic manner in previous layers, the input to the last layer can be set to have the size of 130x130, and that the output of that last layer can be set to have the size of 128x128. As an example, if the eight last layers are 3x3 conv2D layers, and the input to the first of these is 144x144, by using padding=(0,0) in all eight layers, the output of the first layer would have the size of 142x142, the output of the second layer would have the size of 140x140, the output of the third layer would have the size of 138x138, ... all the way to the output of the eight and last layer which would have the size of 128x128. The complexity would then have been reduced from (144*144*Cin*Cout)*8 to (142*142+140*140+138*138+136*136+ 134*134+132*132+130*130+128*128)*(Cin*Cout), which is a complexity reduction by 12%. If there are more than eight layers in the neural network, padding=(l,l) can be used to keep the size intact until the last 8 layers. As an example, if the network consists of 100 layers of 3x3 convolutions, the first 100-8=92 layers can do padding=(l,l) to keep the size at 144x144, but for each of the last eight layers, padding=(0,0) can be done to get to the size of 128x128 at the last layer.

[0108] Cropping by Discarding Data

[0109] Another way to perform a cropping operation, thereby avoiding unnecessary computations is by discarding some of the data before it is input to a layer. This is referred to as “proper cropping” in this disclosure, in order to distinguish it from the zero padding (a.k.a., cropping by padding). Hence, in this disclosure, both “proper cropping” and “cropping by padding” are forms of cropping.

[0110] Let’s assume that the input to the last layer has the size of 144x144 and the output from the last layer has the size of 144x144 before cropping and 128x128 after cropping. If the input to the last layer is cropped from 144x144 to 130x130, the last layer would have the output having the size of 130x130 (this is done by using padding=(l,l)), and after that the middle 128x128 samples can be cropped. This will produce identical results, but the number of multiplications will have been reduced from 144xl44xCinxCout to 130xl30xCinxCout in the last layer, a reduction of -18%.

[0111] It should be noted that cropping by padding is often preferrable to proper cropping. The reason is that with proper cropping, the previous layer uses computational resources to calculate a signal, for instance 144x144x8, but then some of these computed values are simply discarded to get to a signal of, say, size 142x142x8. Cropping by padding instead never calculates these discarded values in the first place. Sometimes however, it may be preferrable to do proper cropping. As an example, in some systems there might be hardware support for doing a conv2D layer with padding=(l,l) but conv2D with padding=(0,0) may be executed in software. In such a situation it is preferable to calculate all values in hardware and then discard some of them.

[0112] Cropping by Padding and by Discarding Data

[0113] A different way to perform a cropping operation, thereby avoiding unnecessary computations is by cropping by padding and discarding some of the data before it is input to a layer. Here is an example of such a combination:

[0114] First, proper cropping can be applied to reduce the patch size from 144x144 to 130x130, and then cropping by padding can be performed (using padding=(0,0)) to get directly to the desired output size of 128x128. This would reduce complexity further from 144xl44xCinxCout to 128xl28xCinxCout, in the last layer, a reduction of -21%.

[0115] It should be noted that cropping, either using ‘proper cropping’ or ‘cropping by discarding’ is different from a resolution change as defined earlier in this document. This means that the patch size before and after the layer is not so that input size / output size = integer or output size / input size = integer.

[0116] In some cases, the input patch size is not a square. As an example, if a patch is filtered at the edge of an image, the patch size may not be 144x144 but instead 144x64. In this case padding=(0,0) of a conv2d with kemel=3x3 will result in an output of 142x62.

[0117] In some cases, the kernel size is not a square such as 3x3 or 5x5, but can have the form of, for instance, 3x1. Often a 3x3 kernel can be approximated by two layers, the first being 3x1 and the second being 1x3. In the 3x1 case, ‘same’ padding is achieved using padding=(l,O), and in the 1x3 case, ‘same’ padding is achieved using padding=(O,l). However, ‘valid’ padding is still obtained using padding=(0,0). Thus, in one aspect, if the input to the second to last layer is 130x130, and the last two layers are 3x1 and 1x3, the padding=(0,0) can be used in the second to last layer to go from 130x130 to 128x130, and padding=(0,0) can be used in the last layer to go from 128x130 to 128x128.

[0118] Reducing the Patch Size Early On

[0119] In some embodiments, the patch size may be reduced in early layers of the neural network architecture and then the patch size is kept intact, for instance, by using padding for the rest of the layers in the NN architecture. For example, the neural network architecture consists of a head, backbone, and tail parts, each with one or more convolutional layers, and according to some embodiments, the patch size can be reduced within the head part and before the backbone part. In this example the patch size is then kept intact through the backbone and tail parts. In another example, the patch size reduction starts in the backbone part and once the patch size has reached the output patch size it is kept intact using, for instance, padding mechanism.

[0120] In some embodiments, the patch size reduction is performed not in every consecutive convolutional layer, but it also includes keeping the patch size intact throughout some of those convolutional layers. In one example the patch size is reduced in convolutional layer L and then maintained through convolutional layer L+l and L+2 and then reduced again in convolutional layer L+3.

[0121] More specifically, given how much complexity can be saved by a smaller size, it may be beneficial to move to the smaller size earlier in the network. For example, let’s assume that the NN filter comprises 100 layers of 3x3 conv2D and that Cin=Cout=l for simplicity. Let’s further assume that the first 92 layers would have a complexity of (144x144) or 144*144*92= 1907712 and the last eight layers a complexity of (142*142+140*140+138*138+136*136+134*134+132*132+130*130+128*128) = 145968. In these embodiments, padding=(0,0)may be applied to the first eight layers. That would mean that the size would go down to 128x128 at the output of the eighth layer. Now the remaining 92 layers can do padding=(l,l) to keep the size at (128x128). The eight first layers would again cost 145968, but the last 92 layers would instead have a much lower cost of 128*128*92 = 1507328. By performing the cropping early, the total complexity would go down from 1907712+145968 to 145968+1507328, a further saving of 20%.

[0122] Note that these embodiments may require a new training of the network in the NN filter or may be able to re-use the same trained weights for the original architecture.

[0123] Spending the Complexity Reduction for Performance Improvement

[0124] In some embodiments, the complexity reduction achieved by the above embodiments is returned to the NN loop filter in the form of increasing the size of the input patch to the neural network in the NN loop filter. In these embodiments, the size of the input patch to the neural network is increased so the neural network takes in a patch having a larger size as compared to the original input patch size. Then, in order for the neural network to provide the same output patch size, the patch size is adjusted through different layers of the neural network, for instance, by proper padding or introducing middle cropping steps. The increased input patch size may increase the computational complexity of the NN loop filter, but it may also improve the performance of the filter.

[0125] In these embodiments, the size of the input patch may be increased up to the point where the total complexity of the operation of the NN filter remains the same. In other words, the increased complexity coming from bigger patch sizes in the earlier layers may be compensated by the reduced complexity caused by the smaller patch sizes in the later layers. Note that these embodiments may require a new training of the neural network or may be able to re-use the same trained weights for the original architecture.

[0126] In some embodiments, the complexity reduction achieved by the above embodiments is returned to the NN loop filter in the form of increasing network parameters of the neural networks in the NN loop filter. Examples of increasing the network parameters include but are not limited to adding more layers to the network, increasing the kernel size in one or more layers, etc. Note that, in these embodiments, retraining the neural networks in the NN loop filter may be required either via a complete retraining or a fine-tuning step.

[0127] As an example, let’s assume that the original network of the NN loop filter has a complexity of 17 kMAC / pixel. Let’s further assume that after applying the conceptof the above embodiments, the complexity is reduced to 16 kMAC / pixel with equal or similar performance results. In case the complexity of 17 kMAC / pixel is a permitted complexity, then there is an extra budget of 1 kMAC / pixel to make the network deeper, wider, or have more branches, or otherwise make the network bigger. This deepening or widening of the network will, after appropriate training, produce a better BD-rate saving than the original network at the same complexity. This way, the gains saved in complexity can instead be traded to gains in quality measured in BD-rate.

[0128] Increasing Kernel Size and Patch Size of Intermediate Layers

[0129] In some embodiments, the size of the kernel of one or more intermediate convolutional layers may be increased with a larger padding size. This would increase the patch size. Then, in the subsequent layers or after several layers, a gradual padding can be applied to crop the larger patch, thereby reducing the size to the original patch size.

[0130] Reducing Patch Size Within Residual Blocks

[0131] As explained above, in the NN loop filter shown in FIG. 4, in case the output 405 of the last back bone block has the patch size of 72x72, the patch size of the output 410 of the CONV 3x3 layer can be reduced to 68x68 by processing the output 405 using the CONV 1x3, CONV 3x1, and CONV 3x3 layers, and thus the output 403 of the Pixel Shuffle layer can have the patch size of 136x136.

[0132] However, in some scenarios, it may be better to further reduce the patch size of the output 403 such that the output 403 have the patch size of 128x128. This is because, as shown in FIG. 16, the output of the NN loop filter is generated by correcting sample data in a portion 1503 of the input picture 1501 with the output 403, and 128x128 is used as the size of the portion 1503.

[0133] In order to further reduce the patch size of the output 403, the output 405 of the last back bone block should be set to be smaller. For example, if the output 405 of the last back bone block is 68x68, then the outputs from the subsequent layers would be as follows:

[0134] As shown in the table above, if the output 405 of the last back bone block is 68x68, then the final residual output 403 of the NN loop filter has the patch size of 128x128, which is exactly the size needed for correcting the sample data in the portion 1503 of the input picture 1501. To reduce the patch size of the output 405 of the last back bone block to 68x68, in some embodiments of this disclosure, one or more of the back bone blocks included in the NN loop filter are modified.

[0135] FIG. 7 shows how a back bone block in the NN loop filter conventionally operates. The back bone block is a residual block. This means that the input 701 of the back bone block is forwarded not only to the first layer of the block (“CONV 1x1 CxCi”) but also to the adder 710.

[0136] The table below shows the size of each of the input 701 and the outputs 702- 709 of the back bone block.

[0137] As shown above, the size of the input 701 and the size of the output 707 are the same - 72x72xC -, which makes it possible to add them together using the adder 710, thereby generating the output 709 which also has the size of 72x72xC. Note that while the number of channels change, the patch size stays the same throughout the back bone block(72x72).

[0138] FIG. 8A shows how a back bone block in the NN loop filter according to some embodiments of this disclosure operates. In FIG. 8 A, the input 801 has the size of 70x70xC instead of 72x72xC. Then the first three layers (“CONV 1x1 CxCi,” “PReLU,” and “CONV 1x1 CixC”) do not change the size, and thus the output 804 of the layer “CONV 1x1 CixC” has the size of 70x70xC.

[0139] However, the next layer “sCONV 1x3 CxC2i” performs a cropping operation by setting the padding to (0,0), which results in the output 805 having the size of 70x68xC (patch size 70x68). This is different from the example shown in FIG. 7. In FIG. 7, because the “sCONV 1x3 CxC2i” layer performs a convolution operation using the kernel with padding = (0,1), the size of the input and the output of the layer are the same. On the contrary, here, because the padding is set to (0,0), the convolution operation of the layer “sCONV 1x3 CxC2i” with the kernel reduces the patch size.

[0140] Similarly, the subsequent layer “sCONV 3x1 C21XC” performs another cropping operation by setting the padding to (0,0), which results in the output 806 having the size of 68x68xC (patch size 68x68). The last layer “sCONV 1x1 CxC” will not change the patch size and thus its output 807 will have the size of 68x68xC.

[0141] In order to add the output 807 to the input 801, in some embodiments, a cropping operation 811 is performed on the input 801. This is because the size of the input 801, which is 70x70xC, does not match the size of the output 807, which is 68x68xC. The cropping operation 811 will remove the border samples (i.e., the top-most, bottom-most, left-most and right-most sample) in every channel so that the resulting output 808 has the size of 68x68xC. The output 808 can be added to the output 807 using the adder 810, thereby producing the output 809, which also has the size of 68x68xC. The patch size of each of the inputs / outputs 801-809 are shown below.

[0142] As explained above, in order for the last back bone block to generate the output 809 having the patch size of 68x68, the input 801 of the block should be 70x70, which means that the output 809 of the second last back bone block should have the size of 70x70. Assuming that the input of the second last back bone block has the size of 72x72xC (i.e., having the patch size of 72x72), the second last back bone block can reduce the patch size in the same way that the last back bone block can reduce the patch size from 70x70 to 68x68. The patch size of each of the inputs / outputs 801-809 of the second last back bone block are shown below.

[0143] Therefore, in case the NN loop filter includes 12 back bone blocks and the input of the first back bone block has the size of 72x72xC, then the first 10 back bone blocks can operate without any cropping operation. The last two back bone blocks can perform the cropping operations reducing the patch size from 72x72 to 68x68. Then, the output from the last back bone block can be provided as the input 405 to the remaining layers of the NN loop filter, thereby generating the output 403 having the size of 128x128x1 as described above.

[0144] In a summary, in the above embodiments, not all back bone blocks of the NN loop filter are architecturally equivalent to each other since some (e.g., the last two backbone blocks) of the blocks may change the patch size while the rest of the blocks (e.g., the first ten back bone blocks) may not change the patch size. This is in contrast to the case where all the back bone blocks have a similar architecture in that none of them changes the patch size.

[0145] As shown in FIG. 8A, the output of the last convolutional layer of a back bone block is combined with the input of the back bone block to generate the output of the back bone block. In order to combine the output of the last convolutional layer with the input of the back bone block, the size of the output of the last convolutional layer and the size of the input of the back bone block should be the same.

[0146] However, as explained above, in case the last two back bone blocks reduces the patch size, the size of the output of the last convolutional layer and the size of the input of the back bone block would be different. In order to solve this problem, in some embodiments of this disclosure, a cropping operation 811 is performed to the input 801. The purpose of the cropping operation 811 is to reduce the size of the input 801 such that the size of the output of the last convolutional layer and the size of the input of the back bone block are the same.

[0147] There are different ways of performing the cropping operation 811.

[0148] In one example, the cropping operation 811 can be performed simply by not forwarding all of the input samples 801 to the adder 810. More specifically, as illustrated in FIG. 8B, the cropping operation 811 can be performed by not forwarding the input samples 801 corresponding to the region 852 to the adder 810 but by forwarding only the input samples 801 corresponding to the region 854.

[0149] In another example, because the above “selective” forwarding of the input samples may not be possible due to some limitations in software, a dummy convolutional layer may be used to simulate the cropping operation. This is illustrated in FIGS. 9A and 9B.

[0150] In FIG. 9A, the input 901 having the size of 70x70xC, which is subject to the cropping operation 911, is fed through a convolution layer 3x3 with C groups. Having C groups means that each of the channels will perform a convolution independently from each other. Here, setting the weights of the kernel to zero except for the middle position, and the bias to 0, and the padding to (0,0) will result in the output 908 having the size of 68x68xC (or (n-2)x(n-2)xC in case the input 901 has the size of nxnxC).

[0151] Another way to simulate this cropping operation is to use two one-dimensional convolution layers, as shown in FIG 9B. In FIG. 9B, the input 951 having the size of (n+2)x(n+2)xC is first fed to a 1x3 convolutional layer with C groups and having padding=(0,0). By setting the weights of the kernel of this convolutional layer to [0, 1, 0] and the bias term to 0, it is possible to simulate a cropping operation so that the output 970 has the size of size (n+2)xnxC. This output is then fed to a 3x1 convolutional layer with C groups and having padding = (0,0). By setting the weights of the kernel of this convolutional layer toO'1.0. and the bias term to 0, it is possible to simulate another cropping operation so that the output 958 of a cropping operation 961 has the size of nxnxC.

[0152] In some cases, it may be possible to further improve the performance of the filter by doing the cropping operations of the bypass link using convolutions, but allowing non-zero weight outside the center, as well as allowing the middle weight to be different from 1.

[0153] FIG. 10A shows a process 1000 for processing an image, according to some embodiments. The process 1000 may begin with step S1002. The step sl002 comprises selecting an input patch within the image. The input patch has a patch size (Yin, Xin) and Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction. Step si 004 comprises processing the input patch using a neural network, NN, filter, thereby generating an output patch. Step si 006 comprises modifying the image based at least on the output patch. The NN filter comprises convolutional layers Ci-Cn arranged in a sequence. Processing the input patch using the NN filter comprises outputting from the convolutional layer Ck a first patch having a patch size of (Yl, XI). k is equal to or less than n, and Yl is a length of the first patch along the first direction and XI is a length of the first patch along the second direction. Yl = al x Yin, XI = bl x Xin, each of Yl, XI, Yin, and Xin is an integer that is greater than or equal to 1, and both al and 1 / al are nonintegers and / or both bl and 1 / bl are non-integers.

[0154] In some embodiments, processing the input patch using the NN filter further comprises obtaining a second patch; providing the second patch to the convolutional layer Ck; and generating the first patch using the convolutional layer Ck by applying a convolution operation to the second patch. The second patch has a patch size of (Y2, X2), Y2 is a length of the second patch along the first direction and X2 is a length of the second patch along thesecond direction, Y1 = a2 x Y2, XI = b2 x X2, each of Y2 and X2 is an integer that is greater than or equal to 1, and both a2 and l / a2 are non-integers and / or both b2 and l / b2 are non-integers.

[0155] In some embodiments, the convolution operation is applied to the second patch using a kernel having a size (Yk, Xk) with padding having a size of (Yp, Xp), floor(Yk / 2) > Yp and / or floor(Xk / 2) > Xp, each of Yk and Xk is an integer that is greater than or equal to 1, and each of Yp and Xp is an integer that is greater than or equal to 0.

[0156] In some embodiments, processing the input patch using the NN filter further comprises obtaining a second patch; cropping the second patch by removing samples from a border of the second patch, thereby generating a cropped second patch; providing the cropped second patch to the convolutional layer Ck; and generating the first patch using the convolutional layer Ck by applying a convolution operation to the cropped second patch. The second patch has a patch size of (Y2, X2), Y2 is a length of the second patch along the first direction and X2 is a length of the second patch along the second direction, Y1 = a2 x Y2, XI = b2 x X2, each of Y2 and X2 is an integer that is greater than or equal to 1, and both a2 and l / a2 are non-integers and / or both b2 and l / b2 are non-integers.

[0157] In some embodiments, the cropped second patch has a size of (Y2’, X2’), Y2’ is a length of the cropped second patch along the first direction and X2’ is a length of the cropped second patch along the second direction, the convolution operation is applied to the cropped second patch using a kernel having a size (Yk, Xk) with padding having a size of (Yp, Xp), Y2’-Y1 = 2* (floor(Yk / 2)-Yp), X2’-X1 = 2* (floor(Xk / 2)-Xp), each of Yk and Xk is an integer that is greater than or equal to 1, and each of Yp and Xp is an integer that is greater than or equal to 0.

[0158] In some embodiments, Y2’=Y1 and X2’=X1.

[0159] In some embodiments, Y2’>Y1 and X2’>X1.

[0160] In some embodiments, k is less than n, and obtaining the second patch comprises obtaining a third patch; providing the third patch to the convolutional layer Ck-i, thereby generating the second patch.

[0161] In some embodiments, Y2-Y1 is prefixed and is independent of the value of Y2, and X2-X1 is prefixed and is independent of the value of X2.

[0162] In some embodiments, the convolutional layers comprise a first set of one or more convolutional layers and a second set of one or more convolution layers, and processing the input patch using the NN filter comprises cropping, by removing samples from a patchborder, a patch to be inputted to each convolutional layer included in the first set and providing the cropped patch to each convolutional layer included in the first set; and / or cropping, by using a convolution operation, a patch that is inputted to each convolutional layer included in the second set. The first and second sets are different or the same.

[0163] In some embodiments, the NN filter comprises a head part, a backbone part, and a tail part, which are arranged in a sequence, and the first and second sets of the convolutional layers are included only in the head part, only in the backbone part, or only in the tail part.

[0164] In some embodiments, the NN filter comprises a head part, a backbone part, and a tail part, and the first and second sets of the convolutional layers are included in two or more of the head part, the backbone part, and the tail part.

[0165] In some embodiments, the NN filter comprises one or more processing blocks arranged in a sequence, and a size of a patch inputted to at least one of said one or more processing blocks is different from a size of a patch outputted from said at least one of said one or more processing blocks.

[0166] In some embodiments, said one or more processing blocks include a processing block R including the first set of one or more convolutional layers and / or the second set of one or more convolutional layers, processing the input patch using the NN filter comprises: obtaining an input patch PR-in; processing the input patch PR-in via the first and second sets of one or more convolutional layers included in the processing block R, thereby generating an output patch PR-out; providing the output patch PR-out to an adder; cropping the input patch PR- in, thereby generating a cropped input patch PR-in-croPPed; forwarding the cropped input patch PR-in-croPPed to the adder; and using the adder, adding the output patch Pa-out to the cropped input patch Pa-in-croPPed, thereby generating a combined patch. The output patch is generated based on the combined patch.

[0167] In some embodiments, cropping the input patch Pa-in is performed by not forwarding to the adder all samples included in the input patch Pa-in.

[0168] In some embodiments, cropping the input patch Pa-in is performed by using at least one convolutional layer with a kernel having a size (Yk, Xk) without any padding or with padding having a size of (Yp, Xp), where Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

[0169] In some embodiments, all weights of the kernel are zero except for a middle weight, and bias term of said at least one convolutional layer is zero.

[0170] In some embodiments, the input patch PR-in has a size of (YR-IH, XR-IH), YR-IH is alength of the input patch PR-in along the first direction and XR-in is a length of the input patch PR-in along the second direction, the output patch PR-out has a size of (Y R- out, XR- out), YR-out is a length of the output patch PR-out along a first direction and XR-out is a length of the output patch PR-out along a second direction, YR-out = a3 X YR-IU, XR- out = b3 x XR-IU, each of YR-IU, XR-in, YR- out, and XR- out is an integer that is greater than or equal to 1, and both a3 and l / a3 are non-integers and / or both bl and 1 / bl are non-integers.

[0171] In some embodiments, the NN filter is a loop-filter in an image or video encoder or decoder.

[0172] In some embodiments, said one or more processing blocks are residual blocks included in the image or video encoder or decoder.

[0173] In some embodiments, the output patch contains reconstructed samples.

[0174] FIG. 10B shows a process 1050 for processing an image. The process 1050 may begin with step S1502. The step s!502 comprises selecting an input patch within the image, wherein the input patch has a patch size. Step si 504 comprises processing the input patch using a neural network, NN, filter, thereby generating an output patch. Step si 506 comprises modifying the image based at least on the output patch. The NN filter comprises processing blocks arranged in a sequence, the processing blocks comprise a processing block R, the processing block R comprises layers LI -Ln arranged in a sequence, processing the input patch using the NN filter comprises inputting a first patch to the processing block R, wherein the inputted first patch is provided to the layer LI of the processing block R; cropping the inputted first patch, thereby generating a cropped first patch and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch. A size of the cropped first patch is smaller than a size of the first patch.

[0175] In some embodiments, cropping the inputted first patch is performed by not forwarding to the adder all samples included the first patch inputted to the processing block R.

[0176] In some embodiments, cropping the inputted first patch is performed by using at least one convolutional layer with a kernel having a size (Yk, Xk) with padding having a size of (Yp, Xp), and Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

[0177] In some embodiments, all weights of the kernel are zero except for a middle weight, and bias term of said at least one convolutional layer is zero.

[0178] FIG. 11 is a block diagram of an apparatus 1100 for implementing the encoder112 and / or the decoder 114, according to some embodiments. When apparatus 1100 implements a decoder, apparatus 1100 may be referred to as a “decoding apparatus 1100,” and when apparatus 1100 implements an encoder, apparatus 1100 may be referred to as an “encoding apparatus 1100.” As shown in FIG. 11, apparatus 1100 may comprise: processing circuitry (PC) 1102, which may include one or more processors (P) 1155 (e.g., a general purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., apparatus 1100 may be a distributed computing apparatus); at least one network interface 1148 comprising a transmitter (Tx) 1145 and a receiver (Rx) 1147 for enabling apparatus 1100 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 1148 is connected (directly or indirectly) (e.g., network interface 1148 may be wirelessly connected to the network 110, in which case network interface 1148 is connected to an antenna arrangement); and a storage unit (a.k.a., “data storage system”) 1108, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1102 includes a programmable processor, a computer program product (CPP) 1141 may be provided. CPP 1141 includes a computer readable medium (CRM) 1142 storing a computer program (CP) 1143 comprising computer readable instructions (CRI) 1144. CRM 1142 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1144 of computer program 1143 is configured such that when executed by PC 1102, the CRI causes apparatus 1100 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 1100 may be configured to perform steps described herein without the need for code. That is, for example, PC 1102 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0179] The core concept of some embodiments of this disclosure can be summarized as follows.

[0180] In processing an image, neural network(s) (“NN(s)”) may be used. The NN(s) used for processing an image may be referred to as an NN filter. In one example, the NN filter is an NN loop filter included in an image / video encoder and can be used for encodingan image / video. In another example, the NN filter is an NN loop filter included in an image / video decoder and can be used for decoding an image / video. FIG. 17A shows a simplified view of an exemplary structure of an NN filter 1700.

[0181] As shown in FIG. 17A, the NN filter 1700 comprises a plurality of convolutional layers 1702, 1704, 1706, 1708, ... , arranged in a sequence. In the sequence of the convolutional layers, the output of a convolutional layer is provided as an input to another convolutional layer. For example, in FIG. 17A, the output of the layer 1702 is provided as an input to the layer 1704, the output of the layer 1704 is provided as an input to the layer 1706, ... Note that there may be one or more non-convolutional layers between any two adjacent convolutional layers. One example of such non-convolutional layers is parametric rectified linear unit (PReLU). Also note that not all convolutional layers included in the NN filter 1700 are arranged in a sequence. For example, the NN filter 1700 may comprise one or more convolutional layers (e.g., the convolutional layer 1710) that are connected to other convolutional layer(s) (e.g., the layer 1702) in parallel.

[0182] In the NN filter 1700, a patch may be provided to each of the plurality of convolutional layers as an input and each convolutional layer may perform a convolution operation on the inputted patch using a kernel. As known in the art, the convolution operation includes performing mathematical operations such as multiplications and additions on sample values of a patch inputted to each convolutional layer using values of a kernel.

[0183] For example, FIG. 18A shows an input patch 1802 inputted to a convolutional layer and a kernel 1804 of the convolutional layer. As shown in FIG. 18A, the input patch 1802 comprises sample values Pl 1, P12, P13, P21, P22, P23, P31, P32, and P33 and the kernel 1804 comprises values Kl l, K12, K21, and K22. A convolution operation is performed on the input patch 1802 by moving the kernel 1804, for example, as shown in FIGS. 18B-E and applying the values of the kernel to the sample values. For example, the sample value 011 of the output patch 1806 is obtained by applying the kernel 1804 to the top left four pixels of the input patch 1802 as shown in FIG. 18B, the sample value 012 of the output patch 1806 is obtained by applying the kernel 1804 to the top right four pixels as shown in FIG. 18C, the sample value 021 of the output patch 1806 is obtained by applying the kernel 1804 to the bottom left four pixels as shown in FIG. 18D, and the sample value 022 of the output patch 1806 is obtained by applying the kernel 1804 to the bottom rightfour pixels as shown in FIG. 18E. A bias value B may also be associated with the kernel; this bias value may be added after the convolution. (If there are more output channels, there may be more than one bias value.) The sample values of the output patch 1806 can be calculated as follows: Oi l = Pll x Kl l + P12 x K12 + P21 x K21 + P22 x K22 + B, 012 = P12 x Kll + P13 x K12 + P22 x K21 + P23 x K22 + B, 021 = P21 x Kl l + P22 x K12 + P31 x K21 + P32 x K22 + B, and 022 = P22 x Kl l + P23 x K12 + P32 x K21 + P33 x K22 + B.

[0184] The operational complexity of a convolutional layer is proportional to the number of the multiplications and additions required to perform a convolution operation of the convolutional layer, and the number of the multiplications and additions required to perform the convolution operation is proportional to the size of the input patch 1802. Thus, in order to reduce the operational complexity of the convolutional layer(s) included in the NN filter, the size of an input patch to be provided to a convolutional layer may be reduced. This size reduction may be referred to as a cropping operation.

[0185] There are two different ways of performing the cropping operation for a convolutional layer. The first way is by removing sample values from the border of a patch to be inputted to the convolutional layer and the second way is by performing, at a previous convolution layer and / or at the present convolution layer (that is the convolution layer to which the patch is to be inputted), a convolutional operation without padding or with reduced padding.

[0186] For example, let’s assume that there is an input patch 1712 having a size (NxM) which is to be inputted to the convolutional layer 1708 where N and M may be the same or different. In order to reduce the complexity of the convolution operation performed by the convolutional layer 1708, the size of the input patch 1712 needs to be reduced before being inputted to the convolutional layer 1708.

[0187] The first way to reduce the size of the input patch 1712 is by providing, as shown in FIG. 17B, a separate cropping layer 1720 between the previous convolution layer 1706 and the convolution layer 1708. This separate cropping layer 1720 is not a convolutional layer. The cropping layer 1720 is configured to receive the input patch 1712, remove sample values from the border of the input patch 1712, thereby generating a cropped input patch 1722 having a size (N-Cl x M-C2) where (Cl > 0 and C2 >= 0), (Cl >=0 and C2 > 0), or (Cl>0 and C2>0) where Cl corresponds to the number of croppedpixels along a first direction (e.g., y-direction) and C2 corresponds to the number of cropped pixels along a second direction (e.g., x-direction).

[0188] The second way is to reduce the size of the input patch 1712 by performing a cropping operation without padding (i.e., with padding (0,0)) at the previous convolutional layer 1704. For example, as shown in FIG. 18A, when a convolution operation is performed on a patch without padding, the output of the convolution operation is a patch having a size that is smaller than the size of the patch that went through the convolution operation. For example, in FIG. 18 A, the size of the input patch 1802 is 3x3 while the size of the output patch 1806 is 2x2. Thus, by performing a convolution operation at any one or more of the previous convolutional layers 1702, 1704, and 1706, the size of the input patch 1712 to be provided to the convolutional layer 1708 can be reduced.

[0189] In some embodiments, a cropping operation may be achieved by performing a convolution operation on a patch even with padding. This is illustrated in FIGS. 19A, 19B, 20 A, and 20B.

[0190] FIG. 19A shows performing a convolution operation on a 5x5 input patch 5x5 using a 4x4 kernel without padding. As shown in FIG. 19B, the convolution operation is performed by moving the kernel until there is no sample values in the 5x5 input patch that were not processed by the kernel. This convolution operation would result in an 2x2 output patch (corresponding to the number of times that the kernel has moved). Here, the size difference between the input patch and the output patch is (5-2, 5-2) = (3,3).

[0191] FIG. 20A shows performing the same convolution operation shown in FIG. 19A but with padding (1,1). The area where the padding is applied corresponds to squares with “P.” Note that “P” does not indicate any specific value. Rather it merely indicates that the padding is applied to the area. As shown in FIG. 20A, applying the padding (1,1) to the 5x5 input patch would result in a 7x7 patch. As shown in FIG. 20B and FIG. 21, the convolution operation is performed by moving the kernel until there is no sample values in the 7x7 patch that were not processed by the kernel. This convolution operation would result in an 4x4 output patch (corresponding to the number of times that the kernel has moved). Here, the size difference between the input patch and the output patch is (5-4, 5-4) = (1,1). This shows that even when the non-zero padding is applied, in case floor(Ky / 2) > Py and / or floor( Kx / 2) > Px), a cropping operation is performed by merely performing a convolution operation.

[0192] In some embodiments, one or more convolution layers, e.g., 1702-1708, included in the NN filter 1700 are included in a processing block of the NN filter 1700, which includes a feed forward path. For example, FIG. 22 shows a simplified structure of a processing block (e.g., a residual block) 2200. As shown in FIG. 22, the output 2202 of the processing block 2200 is determined based on a sum of the output 2204 the last convolutional layer (e.g., 1708 in FIG. 22) included in the processing block 2100 and the input 2206 of the processing block 2200. Note that there may be one or more additional layers (for processing the output 2204 of the last convolutional layer 1708) between the last convolution layer 1708 and an adder 2212. However, for simple explanation purpose, in this example, it will be assumed that the output 2204 of the last convolution layer 1708 is provided to the adder 2212.

[0193] In case a patch to be inputted to any one or more of the convolution layer 1702-1708 is cropped as described above, the size of the input 2206 of the processing block 2200 and the size of the output 2204 of the last convolutional layer 1708 would be different. More specifically, the size of the output 2204 of the last convolutional layer 1708 would be smaller than the size of the input 2206 of the processing block 200. Thus, according to some embodiments, an additional cropping operation may be performed on the input 2206 of the processing block 2200 in the feed forward path such that the size of the cropped input 2206’ forwarded to the adder 2212 is same as the size of the cropped output 2204. The cropping of the input may be performed by a convolutional layer or a separate cropping layer, as described above.

[0194] Summary of Embodiments

[0195] In contrast to the existing art, in the proposed solution the size of the output patch from the last layer of the NN loop filter is different from the size of the input patch to the NN loop filter.

[0196] In some embodiments of the proposed solution, the size difference between the input and the output patch size is fixed, hence independent from the input patch size and variation of the input patch size.

[0197] In some embodiments of the proposed solution, the size of the input and the size of the output are not multiples of each other, meaning that input size / output size is not an integer, and output size / input size is also not an integer.

[0198] In the proposed method, there is at least one resizing step between two layers of the neural network where this resizing step may include a cropping or a zero padding.

[0199] In some embodiments of the proposed solution no further cropping step exists after the last layer of the NN loop filter.

[0200] In some embodiments of the proposed solution all calculations in the neural network affect the value of the useful pixels in the output patch of the last layer of the neural network.

[0201] In some embodiments of the proposed method the output patch size is already achieved in one of the middle layers of the NN loop filter. By a middle layer we here mean any layer that is not the first or the last layer.

[0202] In some embodiments of the proposed methods the patch size reduction is performed in more than one step, for instance a patch size reduction of 10 pixels is performed in 5 different convolutional layers each with a 3x3 kernel and padding of zero pixels in the borders. In this example each convolutional layer will reduce the patch size by 2 pixels in X and 2 pixels in Y dimension.

[0203] In some embodiments of the proposed method a neural network model is trained with equal input and output patch sizes as described in the existing art, but the inference step is then performed using the proposed model with output patch size smaller than the input patch size. In other words, the training of the existing art models may be reused for a model following the proposed method.

[0204] In another embodiment the complexity reduction by the proposed methos is traded back into the model by increasing the model parameters, such as making the model deeper, wider or increasing kernel sizes, increasing patch size etc., in order to improve the performance of the model.

[0205] In some embodiments of the proposed solution the input patch size in larger and the output patch size is smaller than the input and output patch sizes for the NN filter in the existing art.

[0206] In some embodiments of the proposed solution there is a cropping step in the feed forward path in the NN architecture.

[0207] Additional Summary of Embodiments

[0208] Al. A method (1000) for processing an image, the method comprising: selecting (si 002) an input patch within the image, wherein the input patch has a patch size (Yin, Xin) and further wherein Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction; processing (si 004) the input patch using a neural network, NN, filter, thereby generating an output patch; and modifying (si 006) the image based at least on the output patch, wherein the NN filter comprises convolutional layers Ci-Cn arranged in a sequence (that is, the convolutional layers are arranged in a sequential sequence starting with convolutional layer Ci and ending with convolutional layer Cn such that, for i = 1 to n, convolutional layer Ci immediately follows convolutional layers Ci-i in the sequence and immediately precedes convolutional layers Ci+i in the sequence), processing the input patch using the NN filter comprises: outputting from the convolutional layer Ck a first patch having a patch size of (Yl, XI), wherein k is equal to or less than n, and further wherein Yl is a length of the first patch along the first direction and XI is a length of the first patch along the second direction, Yl = al x Yin, XI = bl x Xin, each of Yl, XI, Yin, and Xin is an integer that is greater than or equal to 1, and both al and 1 / al are nonintegers and / or both bl and 1 / bl are non-integers.

[0209] Cropping at the convolutional layer

[0210] A2. The method of embodiment Al, wherein processing the input patch using the NN filter further comprises: obtaining a second patch; providing the second patch to the convolutional layer Ck; and generating the first patch using the convolutional layer Ck by applying a convolution operation to the second patch, the second patch has a patch size of (Y2, X2), Y2 is a length of the second patch along the first direction and X2 is a length of the second patch along the second direction, Yl = a2 x Y2, XI = b2 x X2, each of Y2 and X2 is an integer that is greater than or equal to 1, and both a2 and l / a2 are non-integers and / or both b2 and l / b2 are non-integers.

[0211] Cropping at the convolutional layer

[0212] A3. The method of embodiment A2, wherein the convolution operation is applied to the second patch using a kernel having a size (Yk, Xk) with padding having a size of (Yp, Xp), floor(Yk / 2) > Yp and / or floor(Xk / 2) > Xp, each of Yk and Xk is an integer that is greater than or equal to 1, and each of Yp and Xp is an integer that is greater than or equal to 0.

[0213] Cropping before the convolutional layer

[0214] A4. The method of embodiment Al, wherein processing the input patch using the NN filter further comprises: obtaining a second patch; cropping the second patch by removing samples from a border of the second patch, thereby generating a cropped second patch; providing the cropped second patch to the convolutional layer Ck; and generating the first patch using the convolutional layer Ck by applying a convolution operation to the cropped second patch, the second patch has a patch size of (Y2, X2), Y2 is a length of the second patch along the first direction and X2 is a length of the second patch along the second direction, Y1 = a2 x Y2, XI = b2 x X2, each of Y2 and X2 is an integer that is greater than or equal to 1, and both a2 and l / a2 are non-integers and / or both b2 and l / b2 are non-integers.

[0215] Cropping before the convolutional layer and no cropping at the convolutional layer OR Cropping before the convolutional layer and additional cropping at the convolutional layer

[0216] A5. The method of embodiment A4, wherein the cropped second patch has a size of (Y2’, X2’), Y2’ is a length of the cropped second patch along the first direction and X2’ is a length of the cropped second patch along the second direction, the convolution operation is applied to the cropped second patch using a kernel having a size (Yk, Xk) and with padding having a size of (Yp, Xp), Y2’-Y1 = 2* (floor(Yk / 2)-Yp), X2’-X1 = 2* (floor(Xk / 2)-Xp), each of Yk and Xk is an integer that is greater than or equal to 1, and each of Yp and Xp is an integer that is greater than or equal to 0.

[0217] Cropping before the convolutional layer and no cropping at the convolutional layer

[0218] A6. The method of embodiment A5, wherein Y2’=Y1 and X2’=X1.

[0219] Cropping before the convolutional layer and additional cropping at the convolutional layer

[0220] A7. The method of embodiment A5, wherein Y2’>Y1 and X2’>X1.

[0221] A8a. The method of any one of embodiments A2-A7, wherein k is less than n, and obtaining the second patch comprises: obtaining a third patch; providing the third patch to the convolutional layer Ck-i, thereby generating the second patch.

[0222] A8b. The method of embodiment A8a, wherein the third patch is generated by convolutional layer Ck-2 , k > 2.

[0223] A9. The method of any one of embodiments A2-A8b, wherein Y2-Y1 isprefixed and is independent of the value of Y2, and / or X2-X1 is prefixed and is independent of the value of X2.

[0224] A10. The method of any one of embodiments A1-A9, wherein the convolutional layers comprise a first set of one or more convolutional layers and a second set of one or more convolution layers, and processing the input patch using the NN filter comprises: cropping, by removing samples from a patch border, a patch to be inputted to each convolutional layer included in the first set and providing the cropped patch to each convolutional layer included in the first set; and / or cropping, by using a convolution operation, a patch that is inputted to each convolutional layer included in the second set, and the first and second sets are different or the same.

[0225] Al l. The method of embodiment A10, wherein the NN filter comprises a head part, a backbone part, and a tail part, which are arranged in a sequence, and the first and second sets of the convolutional layers are included only in the head part, only in the backbone part, or only in the tail part.

[0226] Al 2. The method of embodiment A10, wherein the NN filter comprises ahead part, a backbone part, and a tail part, and the first and second sets of the convolutional layers are included in two or more of the head part, the backbone part, and the tail part.

[0227] Al 3. The method of any one of embodiments A10-A12, wherein the NN filter comprises one or more processing blocks arranged in a sequence, and a size of a patch inputted to at least one of said one or more processing blocks is different from a size of a patch outputted from said at least one of said one or more processing blocks.

[0228] Al 4. The method of embodiment Al 3, wherein said one or more processing blocks include a processing block R including the first set of one or more convolutional layers and / or the second set of one or more convolutional layers, processing the input patch using the NN filter comprises: obtaining an input patch PR-in; processing the input patch PR-in via the first and second sets of one or more convolutional layers included in the processing block R, thereby generating an output patch PR-out; providing the output patch PR-out to an adder; cropping the input patch PR-in, thereby generating a cropped input patch PR-in-croPPed; forwarding the cropped input patch PR-in-croPPed to the adder; and using the adder, adding the output patch PR-out to the cropped input patch PR-in-croPPed, thereby generating a combined patch, and the output patch is generated based on the combined patch.

[0229] Al 4a. The method of embodiment Al 4, wherein cropping the input patch PR-IU is performed by not forwarding to the adder all samples included in the input patch PR-IU.

[0230] A14b. The method of embodiment A14, wherein cropping the input patch PR-IU is performed by using at least one convolutional layer with a kernel having a size (Yk, Xk) without any padding or with padding having a size of (Yp, Xp), where Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

[0231] Al 4c. The method of embodiment Al 4b wherein all weights of the kernel are zero except for a middle weight, and bias term of said at least one convolutional layer is zero.

[0232] A15. The method of any one of embodiments A14-A14c, wherein the input patch PR-in has a size of (Y R-in, XR-in), YR-IH is a length of the input patch PR-IU along the first direction and XR-in is a length of the input patch PR-IU along the second direction, the output patch PR-out has a size of (Y R- out, XR- out), YR-out is a length of the output patch PR-out along a first direction and XR-out is a length of the output patch PR-out along a second direction, YR-OUI = a3 x YR-in, XR- out = b3 x XR-IU, each of YR-IU, XR-IU, YR- out, and XR- out is an integer that is greater than or equal to 1, and both a3 and l / a3 are non-integers and / or both bl and 1 / bl are non-integers.

[0233] Al 6. The method of any one of embodiments Al -Al 5, wherein the NN filter is a loop-filter in an image or video encoder or decoder.

[0234] Al 7. The method of embodiment Al 6 when embodiment Al 6 depends on any one of embodiments A13-A15, wherein said one or more processing blocks are residual blocks included in the image or video encoder or decoder.

[0235] Al 8. The method of embodiment Al 6 or Al 7, wherein the output patch contains reconstructed samples.

[0236] Bl. A method (1050) for processing an image, the method comprising: selecting (si 052) an input patch within the image, wherein the input patch has a patch size; processing (si 054) the input patch using a neural network, NN, filter, thereby generating an output patch; and modifying (si 056) the image based at least on the output patch, wherein the NN filter comprises processing blocks arranged in a sequence, the processing blocks comprise a processing block R, the processing block R comprises layers LI -Ln arranged in a sequence, processing the input patch using the NN filter comprises: inputting a first patch to the processing block R, wherein the inputted first patch is provided to the layer LI of theprocessing block R; cropping the inputted first patch, thereby generating a cropped first patch and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch, and a size of the cropped first patch is smaller than a size of the first patch.

[0237] B2. The method of embodiment Bl, wherein cropping the inputted first patch is performed by not forwarding to the adder all samples included in the first patch inputted to the processing block R.

[0238] B3. The method of embodiment Bl, wherein cropping the inputted first patch is performed by using at least one convolutional layer with a kernel having a size (Yk, Xk) with padding having a size of (Yp, Xp), and Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

[0239] B4. The method of embodiment B3, wherein all weights of the kernel are zero except for a middle weight, and a bias term of said at least one convolutional layer is zero.

[0240] Cl. A computer program (1100) comprising instructions (1144) which when executed by processing circuitry (1102) cause the processing circuitry to perform the method of any one of embodiments A1-B4.

[0241] C2. A carrier containing the computer program of embodiment Cl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0242] DI. An apparatus (1100) for processing an image, the apparatus being configured to: select (si 002) an input patch within the image, wherein the input patch has a patch size (Yin, Xin) and further wherein Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction; process (si 004) the input patch using a neural network, NN, filter, thereby generating an output patch; and modify (si 006) the image based at least on the output patch, wherein the NN filter comprises convolutional layers Ci-Cn arranged in a sequence, processing the input patch using the NN filter comprises: outputting from the convolutional layer Ck a first patch having a patch size of (Yl, XI), wherein k is equal to or less than n, and further wherein Y1 is a length of the first patch along the first direction and XI is a length of the first patch along the second direction, Yl = al x Yin, XI = bl x Xin, each of Yl, XI, Yin, and Xin is an integer that is greater than or equal to 1, and both al and 1 / al are non-integers and / or both bl and 1 / bl arenon-integers.

[0243] D2. The apparatus of embodiment DI, wherein the apparatus is configured to perform the method of any one of embodiments A2-A18.

[0244] El . An apparatus (1100) for processing an image, the apparatus being configured to: select (si 052) an input patch within the image, wherein the input patch has a patch size; process (si 054) the input patch using a neural network, NN, filter, thereby generating an output patch; and modify (si 056) the image based at least on the output patch, wherein the NN filter comprises processing blocks arranged in a sequence, the processing blocks comprise a processing block R, the processing block R comprises layers LI -Ln arranged in a sequence, processing the input patch using the NN filter comprises: inputting a first patch to the processing block R, wherein the inputted first patch is provided to the layer LI of the processing block R; cropping the inputted first patch, thereby generating a cropped first patch and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch, and a size of the cropped first patch is smaller than a size of the first patch.

[0245] E2. The apparatus of embodiment El, wherein the apparatus is configured to perform the method of any one of embodiments B2-B4.

[0246] FL An apparatus (1100) comprising: processing circuitry (1102); and a memory (1141), said memory containing instructions executable by said processing circuitry, whereby the apparatus is operative to perform the method of any one of embodiments A1-B4.

[0247] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0248] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e., one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender orindirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node). Further, as used herein “a” means “at least one” or “one or more.”

[0249] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

[0250] Reference List

[0251] [1] P. Cerwall (executive editor), et al. Ericsson Mobility Report Nov. 2019(available at: www.ericsson.com / en / mobility-report).

[0252] [2] EE1-1.6: Combined Test of EE1-1.2 and EE1-1.4, Yue Li, Kai Zhang, LiZhang, Hongtao Wang, Jianle Chen, Kevin Reuze, Anand Meher Kotra, Marta Karczewicz , JVET-X0066, Oct. 2021.

[0253] [3] EE1-1.2: Test on Deep In-Loop Filter with Adaptive Parameter Selection and Residual Scaling, Yue Li, Kai Zhang, Li Zhang, Hongtao Wang, Kevin Reuze, Anand Meher Kotra, Marta Karczewicz, JVET-Y0143, Jan. 2022.

[0254] [4] BoG report on NN-filter design unification, E. Alshina, F. Galpin, JVER-AD0380, Apr. 2023.

[0255] [5] BoG Report: Neural Network Technology, A. Segall, JVET-T0130, Oct.2020.

Claims

CLAIMS1. A method (1000) for processing an image, the method comprising: selecting (si 002) an input patch within the image, wherein the input patch has a patch size (Yin, Xin) and further wherein Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction; processing (si 004) the input patch using a neural network, NN, filter, thereby generating an output patch; and modifying (si 006) the image based at least on the output patch, wherein the NN filter comprises convolutional layers Ci-Cn arranged in a sequence, processing the input patch using the NN filter comprises: outputting from the convolutional layer Ck a first patch having a patch size of (Yl, XI), wherein k is equal to or less than n, and further wherein Yl is a length of the first patch along the first direction and XI is a length of the first patch along the second direction,Yl = al x Yin,XI = bl x Xin, each of Yl, XI, Yin, and Xin is an integer that is greater than or equal to 1, and both al and 1 / al are non-integers and / or both bl and 1 / bl are non-integers.

2. The method of claim 1 , wherein processing the input patch using the NN filter further comprises: obtaining a second patch; providing the second patch to the convolutional layer Ck; and generating the first patch using the convolutional layer Ck by applying a convolution operation to the second patch, the second patch has a patch size of (Y2, X2),Y2 is a length of the second patch along the first direction and X2 is a length of the second patch along the second direction,Yl = a2 x Y2,XI = b2 x X2, each of Y2 and X2 is an integer that is greater than or equal to 1, and both a2 and l / a2 are non-integers and / or both b2 and l / b2 are non-integers.

3. The method of claim 2, wherein the convolution operation is applied to the second patch using a kernel having a size (Yk, Xk) and with padding having a size of (Yp, Xp), floor(Yk / 2) > Yp and / or floor(Xk / 2) > Xp, each of Yk and Xk is an integer that is greater than or equal to 1, and each of Yp and Xp is an integer that is greater than or equal to 0.

4. The method of claim 1 , wherein processing the input patch using the NN filter further comprises: obtaining a second patch; cropping the second patch by removing samples from a border of the second patch, thereby generating a cropped second patch; providing the cropped second patch to the convolutional layer Ck; and generating the first patch using the convolutional layer Ck by applying a convolution operation to the cropped second patch, the second patch has a patch size of (Y2, X2),Y2 is a length of the second patch along the first direction and X2 is a length of the second patch along the second direction,Y1 = a2 x Y2,XI = b2 x X2, each of Y2 and X2 is an integer that is greater than or equal to 1, and both a2 and l / a2 are non-integers and / or both b2 and l / b2 are non-integers.

5. The method of claim 4, wherein the cropped second patch has a size of (Y2’, X2’),Y2’ is a length of the cropped second patch along the first direction and X2’ is a length of the cropped second patch along the second direction, the convolution operation is applied to the cropped second patch using a kernel having a size (Yk, Xk) and with padding having a size of (Yp, Xp),Y2’-Y1 = 2* (floor(Yk / 2)-Yp),X2’-X1 = 2* (floor(Xk / 2)-Xp), each of Yk and Xk is an integer that is greater than or equal to 1, andeach of Yp and Xp is an integer that is greater than or equal to 0.

6. The method of claim 5, wherein Y2’=Y1 and X2’=X1.

7. The method of claim 5, wherein Y2’>Y1 and X2’>X1.

8. The method of any one of claims 2-7, wherein k is less than n, and obtaining the second patch comprises: obtaining a third patch; providing the third patch to the convolutional layer Ck-i, k > 1 thereby generating the second patch.

9. The method of claims 8, wherein the third patch is generated by convolutional layer Ck-2 , k > 2.

10. The method of any one of claims 2-9, whereinY2-Y1 is prefixed and is independent of the value of Y2, and / orX2-X1 is prefixed and is independent of the value of X2.

11. The method of any one of claims 1-10, wherein the convolutional layers comprise a first set of one or more convolutional layers and a second set of one or more convolution layers, and processing the input patch using the NN filter comprises: cropping, by removing samples from a patch border, a patch to be inputted to each convolutional layer included in the first set and providing the cropped patch to each convolutional layer included in the first set; and / or cropping, by using a convolution operation, a patch that is inputted to each convolutional layer included in the second set, and the first and second sets are different or the same.

12. The method of claim 11, whereinthe NN filter comprises a head part, a backbone part, and a tail part, which are arranged in a sequence, and the first and second sets of the convolutional layers are included only in the head part, only in the backbone part, or only in the tail part.

13. The method of claim 11, wherein the NN filter comprises a head part, a backbone part, and a tail part, and the first and second sets of the convolutional layers are included in two or more of the head part, the backbone part, and the tail part.

14. The method of any one of claims 11-13, wherein the NN filter comprises one or more processing blocks arranged in a sequence, and a size of a patch inputted to at least one of the one or more processing blocks is different from a size of a patch outputted from the at least one of the one or more processing blocks.

15. The method of claim 14, wherein the one or more processing blocks include a processing block R including the first set of one or more convolutional layers and / or the second set of one or more convolutional layers, processing the input patch using the NN filter comprises: obtaining an input patch PR-in; processing the input patch PR-in via the first and second sets of one or more convolutional layers included in the processing block R, thereby generating an output patch PR-out; providing the output patch PR-out to an adder; cropping the input patch PR-in, thereby generating a cropped input patch PR-in-Cropped^ forwarding the cropped input patch PR-in-croPPed to the adder; and using the adder, adding the output patch PR-out to the cropped input patch PR-in- Cropped, thereby generating a combined patch, and the output patch is generated based on the combined patch.

16. The method of claim 15, wherein cropping the input patch PR-IU is performed by not forwarding to the adder all samples included in the input patch PR-in.

17. The method of claim 15, wherein cropping the input patch PR-IU is performed by using at least one convolutional layer with a kernel having a size (Yk, Xk) without any padding or with padding having a size of (Y p, Xp), where Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

18. The method of claim 17, wherein all weights of the kernel are zero except for a middle weight, and bias term of the at least one convolutional layer is zero.

19. The method of any one of claims 15-18, wherein the input patch PR-IU has a size of (YR-IU, XR-IU),YR-in is a length of the input patch PR-IU along the first direction and XR-in is a length of the input patch PR-in along the second direction, the output patch PR-out has a size of (YR- out, XR- out),YR-out is a length of the output patch PR-out along a first direction and XR-out is a length of the output patch PR-out along a second direction,YR-Out = a3 X YR-In,XR- Out=b3 x XR-in, each of YR-in, XR-IU, YR- out, and XR- out is an integer that is greater than or equal to 1, and both a3 and l / a3 are non-integers and / or both bl and 1 / bl are non-integers.

20. The method of any one of claims 1-19, wherein the NN filter is a loop-filter in an image or video encoder or decoder.

21. The method of claim 20 when claim 20 depends on any one of claims 14-19, wherein the one or more processing blocks are residual blocks included in the image or video encoder or decoder.

22. The method of claim 20 or 21, wherein the output patch contains reconstructed samples.

23. A method (1050) for processing an image, the method comprising: selecting (si 052) an input patch within the image, wherein the input patch has a patch size; processing (si 054) the input patch using a neural network, NN, filter, thereby generating an output patch; and modifying (si 056) the image based at least on the output patch, wherein the NN filter comprises processing blocks arranged in a sequence, the processing blocks comprise a processing block R, the processing block R comprises layers LI -Ln arranged in a sequence, processing the input patch using the NN filter comprises: inputting a first patch to the processing block R, wherein the inputted first patch is provided to the layer LI of the processing block R; cropping the inputted first patch, thereby generating a cropped first patch and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch, and a size of the cropped first patch is smaller than a size of the first patch.

24. The method of claim 23, wherein cropping the inputted first patch is performed by not forwarding to the adder all samples included in the first patch inputted to the processing block R.

25. The method of claim 23, wherein cropping the inputted first patch is performed by using at least one convolutional layer with a kernel having a size (Yk, Xk) with padding having a size of (Yp, Xp), andYp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

26. The method of claim 25, wherein all weights of the kernel are zero except for a middle weight, anda bias term of the at least one convolutional layer is zero.

27. A computer program (1100) comprising instructions (1144) which when executed by processing circuitry (1102) cause the processing circuitry to perform the method of any one of claims 1-26.

28. A carrier containing the computer program of claim 27, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

29. An apparatus (1100) for processing an image, the apparatus being configured to: select (si 002) an input patch within the image, wherein the input patch has a patch size (Yin, Xin) and further wherein Yin is a length of the input patch along a first direction and Xin is a length of the input patch along a second direction; process (si 004) the input patch using a neural network, NN, filter, thereby generating an output patch; and modify (si 006) the image based at least on the output patch, wherein the NN filter comprises convolutional layers Ci-Cn arranged in a sequence, processing the input patch using the NN filter comprises: outputting from the convolutional layer Ck a first patch having a patch size of (Yl, XI), wherein k is equal to or less than n, and further wherein Yl is a length of the first patch along the first direction and XI is a length of the first patch along the second direction,Yl = al x Yin,XI = bl x Xin, each of Yl, XI, Yin, and Xin is an integer that is greater than or equal to 1, and both al and 1 / al are non-integers and / or both bl and 1 / bl are non-integers.

30. The apparatus of claim 29, wherein the apparatus is configured to perform the method of any one of claims 2-22.

31. An apparatus (1100) for processing an image, the apparatus being configured to:select (si 052) an input patch within the image, wherein the input patch has a patch size; process (si 054) the input patch using a neural network, NN, filter, thereby generating an output patch; and modify (sl056) the image based at least on the output patch, wherein the NN filter comprises processing blocks arranged in a sequence, the processing blocks comprise a processing block R, the processing block R comprises layers LI -Ln arranged in a sequence, processing the input patch using the NN filter comprises: inputting a first patch to the processing block R, wherein the inputted first patch is provided to the layer LI of the processing block R; cropping the inputted first patch, thereby generating a cropped first patch and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch, and a size of the cropped first patch is smaller than a size of the first patch.

32. The apparatus of claim 31, wherein the apparatus is configured to perform the method of any one of claims 24-26.

33. An apparatus (1100), the apparatus comprising: memory; and processing circuitry, wherein the apparatus is configured to perform the method of any one of claims 1-26.

Images