Tile early pruning for video encoding and decoding

Abstract

A method of processing an image. The method includes selecting an input tile within the image. The input tile has a tile size (Yin, Xin), Yin being a length of the input tile along a first direction and Xin being a length of the input tile along a second direction. The method also includes processing the input tile using a neural network (NN) filter to generate an output tile. The method also includes modifying the image based on at least the output tile. The NN filter includes convolution layers (C1-Cn) arranged in sequence. Processing the input tile using the NN filter includes: processing the input tile through the convolution layers (C1-Cn) in sequence to generate the output tile. n . Processing the input tile using the NN filter includes: processing the input tile through the convolution layers (C1-Cn) in sequence to generate the output tile. k outputting a first tile having a tile size (Y1, X1). k is less than or equal to n, Y1 is a length of the first tile along the first direction and X1 is a length of the first tile along the second direction, Y1 = a1 x Yin, X1 = b1 x Xin, each of Y1, X1, Yin and Xin is an integer greater than or equal to 1, and a1 and 1 / a1 are both non-integers, and / or b1 and 1 / b1 are both non-integers.

Description

Technical Field

[0001] This disclosure relates to pre-cropping of tiles for video encoding and decoding. Background Technology

[0002] Video compression

[0003] Video is a major form of data service on the network today, and its share is expected to continue to increase (see, for example, reference [1]). One way to reduce data services in video is through compression. In video compression, the source video is encoded into a bitstream, which can then be stored and transmitted to the end user. The end user can use a decoder to extract the video data from the bitstream and display it on the screen. However, since the encoder may not know which device the encoded bitstream will be sent to, the encoder must compress the video into a predetermined format, such as the standardized Universal Video Coding (VVC) format. In this way, all devices that support the selected standard can decode the video. Compression can be lossless, i.e., the decoded video will be the same as the source video provided to the encoder; compression can also be lossy, in which case there is a certain degree of degradation of the acceptable content. Using lossy compression can significantly reduce the bit rate, i.e., the compression ratio can be greatly increased. This is because perfectly reproducing image noise makes lossless compression quite costly.

[0004] A video sequence is a sequence of images. The commonly used color space in video sequences is YCbCr, where Y is the luminance (lightness) component, and Cb and Cr are the chrominance components. Sometimes, the Cb and Cr components are referred to as U and V. Other color spaces are also used. These color spaces include ICtCp, IPT, constant luminance YCbCr, RGB, YCoCg, etc. The embodiments of this disclosure are not limited to a specific color space, but can be applied to a variety of different color spaces.

[0005] VVC and HEVC

[0006] VVC and its predecessor, High Efficiency Video Coding (HEVC), are block-based video codecs developed and standardized jointly by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Moving Picture Experts Group (MPEG). These codecs utilize both temporal and spatial prediction. VVC and HEVC are similar in many ways. Spatial prediction is achieved using intra-frame (I) prediction within the current frame. Temporal prediction is achieved at the block level using one-way (P) or two-way (B) inter-frame prediction from a previously decoded reference frame. In the encoder, the difference between the raw pixel data and the predicted pixel data (called the residual) is transformed to the frequency domain, quantized, and then entropy-coded, and then transmitted along with the necessary prediction parameters (such as prediction mode and motion vectors) that have also been entropy-coded. The decoder performs entropy decoding, inverse quantization, and inverse transform to obtain the residual, which is then added to the intra-frame or inter-frame prediction to reconstruct the image. The VVC version 1 specification was published in 2020 as ITU-T Recommendation H.266 | ISO / IEC 23090-3 “Versatile Video Coding”.

[0007] Blocks and Units

[0008] In many video coding standards (such as HEVC and VVC), each component is divided into multiple blocks, and the encoded video bitstream consists of a series of coded blocks. A block is a two-dimensional array of samples. In video coding, a picture is typically divided into units that cover a specific region of the picture. Each unit consists of all the blocks of all the components that make up that specific region, and each block belongs entirely to one unit. Macroblocks in H.264 and coding units (CUs) in HEVC and VVC are examples of units.

[0009] Alternatively, a block can be defined as a two-dimensional array of transforms used in the applied encoding. These blocks are referred to as "transform blocks". Alternatively, a block can be defined as a two-dimensional array to which a single prediction pattern is applied. These blocks may be referred to as "prediction blocks". In this disclosure, the term "block" is not limited to one of these definitions, but the description herein can be applied to either definition.

[0010] Exploratory Experiments on Neural Network-Based Video Coding (NNVC)

[0011] At the 20th meeting of the Joint Video Experts Group (JVET), it was decided to establish an exploratory experiment (EE) on neural network-based (NN-based) video coding (see, for example, reference [5]). This exploratory experiment continued at the subsequent 21st to 31st JVET meetings, and several tests were conducted: NN-based in-loop filtering, NN-based post-filtering, NN-based super-resolution, and NN-based intra-frame prediction. The first two are particularly relevant to the disclosures below and will be described further below.

[0012] VVC incorporates three non-neural network-based in-loop filters: a deblocking filter, a Sample Adaptive Offset (SAO) filter, and an Adaptive Loop Filter (ALF). The deblocking filter removes block artifacts by smoothing discontinuities across block boundaries in the horizontal and vertical directions. The deblocking filter uses a Block Boundary Strength (BS) parameter to determine the filter strength. BS can have values ​​of 0, 1, and 2, with larger values ​​indicating stronger filtering. The output of the deblocking filter is further processed by the SAO filter, and the output of the SAO filter is subsequently processed by the ALF. The output of the ALF can then be fed into the Decoded Picture Buffer (DPB), which contains the decoded picture that can be used to predict subsequent encoded (or decoded) pictures. Because the deblocking filter, SAO filter, and ALF filter affect the picture used for prediction in the DPB in this way, they are classified as in-loop filters, also known as loop filters. This means that changes made by the loop filter may affect not only the current picture but also future pictures. The decoder can further filter the picture in the DPB but does not store the filtered output in the DPB. Unlike loop filters, this type of filter does not affect future predictions and is therefore classified as a post-processing filter, also known as a post-filter. For decoders, post-filtering is usually optional, so decoder implementations can conform to standard specifications without performing post-filtering.

[0013] Proposals JVET-X0066 and JVET-Y0143 (references [2] and [3], respectively) are two consecutive proposals describing NN-based in-loop filtering. Both proposals use the same NN model for filtering. The NN-based in-loop filter is placed before the SAO filter and ALF, and the samples before the deblocking filter are used as input to this filter. The output of the NN-based in-loop filter is mixed with the output of the deblocking filter and forwarded as input to the SAO filter. The purpose of using NN-based filters is to improve the quality of the reconstructed samples. The nonlinearity of the NN model is advantageous here. The deblocking filter, SAO filter, and ALF all contain nonlinear elements such as conditional statements and are therefore not strictly linear, but all three are based on linear filters. In contrast, a sufficiently large NN model can, in principle, learn any nonlinear mapping and is therefore able to represent a wider range of functions compared to the deblocking filter, SAO filter, and ALF. In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters. In the improved version of this work proposed in proposal JVET-AB0052, only two models are used: one model for luminance samples and another model for chrominance samples.

[0014] JVET-AD0380 (Reference [4]) proposes a new unified design for intra-loop filtering based on neural networks (NNs), which combines the advantages of previous NN structures. This unified filter has only one NN model for filtering luminance and chrominance samples, as well as intra-frame and inter-frame images. Figure 12 The structure of this unified filter is shown. This unified filter has six inputs: the reconstructed luminance and chrominance samples before deblocking (“rec”), the predicted luminance and chrominance samples (“pred”), the BS information for luminance and chrominance (“bs”), the quantization parameters of the sequence (“QPbase”), the quantization parameters for each strip (“QPslice”), and information about whether a particular sample was predicted intra-frame, unidirectionally, or bidirectionally (“IPB”). These inputs are first passed through convolutional layers (3x3 or 1x1) and parameterized rectified linear unit (PReLU) layers, then concatenated and fused together through a 1x1 convolutional layer.

[0015] exist Figure 12 In this code, a convolutional layer with a kernel size of 3x3 is represented as "conv 3x3", and a convolutional layer with a kernel size of 1x1 is represented as "conv 1x1". The values ​​d1, d2, ... represent the number of output channels. "2" indicates downsampling by a factor of 2. There are N backbone blocks (also called trunk blocks). Within a backbone block, a convolutional layer "conv 3x1 CxC21" indicates that the kernel size is 3x1, the number of input channels is C, and the number of output channels is C21.

[0016] The NN-based in-loop filters proposed in JVET-X0066, JVET-AB0053, JVET-AB0052, and JVET-AD0380 significantly improve the compression efficiency of codecs, i.e., they substantially reduce the bit rate without compromising the objective quality measured by MSE-based peak signal-to-noise ratio (PSNR). This improvement in compression efficiency is often simply referred to as "gain" and is typically measured as the Bjontegaard increment rate (BDR) relative to the anchor. For example, a -1% BDR means that the same PSNR distortion can be achieved with an average bit rate saving of 1%. As reported in JVET-AF0041, the BDR for the luma component (Y) is -10.27% for the random access (RA) configuration and -7.86% for the intra-frame (AI) configuration. The complexity of the NN model used for compression is typically measured by MAC / pixel (multiply-accumulate operations per pixel). High bit rate savings with NN models are often directly related to the high complexity of the NN model. The model described in JVET-AF0041 has a complexity of 477kMAC / pixel, meaning 477,000 multiply-accumulate operations are performed per pixel. Other complexity metrics also exist, such as the total size of the model in terms of stored parameters.

[0017] The low-operating-point (LOP) architecture NN loop filter currently used in JVET EE testing (EE1-1.0) is... Figure 13 As shown in (source: JVET-AG2023). In Figure 13 In this model, the input tile size is 144x144, and there is a final cropping step that crops 8 pixels from each side of the output luminance tile and 4 pixels from each side of each chroma tile (which is half the size of the luminance tile). Therefore, in the output, the final cropped tile size is 128x128 for the luminance branch and 64x64 for the chroma branch. The complexity of this current model is approximately 17 kMULT / sample. This means that in order to compute... Figure 13 A sample value in the output of a neural network loop filter architecture requires approximately 17,000 multiplication operations.

[0018] Although Figure 13 The conceptual approach demonstrates how to crop from 144x144 to 128x128, but the implementation in the reference software NNVC version 7 is slightly different: Figure 14 A complete image 1401 (e.g., a 1920x1080 sample) of the brightness sample before the deblocking filter is shown. A 144x144 sample patch 1402 is obtained from this image. This patch contains a 128x128 sample patch 1403 that will be affected by the filtering.

[0019] like Figure 15 As shown, the 144x144 sample patch 1402 is then used as input Rec. EXT Y 1501 is input into the neural network filter. Chroma value 1502, as well as Pred, BS, QPbase, QPSlice, and IPB values, are also input. The neural network processes these inputs and produces a luminance output 1503 with a patch size of 144x144 and a chrominance output 1504 with a patch size of 72x72. For clarity, the following explanation will focus on the luminance output residual 1503. This luminance output residual has a patch size of 144x144, but only the middle 128x128 residual samples are used, and these residual samples are compared with… Figure 14 The 128x128 samples 1403 in image 1401 are added together. Therefore, the addition operation is performed only 128x128 times. In contrast, Figure 12 The addition operation is shown to be performed on a 144x144 sample patch (using a 144x144 pixel rearranged output as the first input, and an input Rec of the same 144x144 patch size). EXT Y is used as the second input), and then cropped to 128x128 only afterwards. The result is the same, but according to... Figure 14 and Figure 15 A consistent coding approach requires only 128 x 128 = 16384 additions, while... Figure 13 It requires 144 x 144 = 20736 additions.

[0020] In the JVET standardization, codecs are developed and studied using Common Test Conditions (CTCs). CTCs specify how the codec under test should be configured and which test sequences should be used. Keeping the test conditions constant allows for comparative evaluation under identical conditions, but its drawback is that configurations outside the CTC are not adequately tested, and the codec's performance may become overly optimized for the CTC.

[0021] The input and output patch sizes of NN-based in-loop filters in existing technologies

[0022] In block-based image and video codecs, the input image is typically divided into multiple blocks, and each block is processed individually throughout the processing pipeline. This makes the codec easily scalable to larger resolutions. To provide a more seamless transition at the boundaries of adjacent blocks and to help the neural network (NN) filter better improve samples near edges, the NN filter is applied to larger pixel patches in the image, and the filter output is then cropped to an appropriate size. In this way, the filtering process for the current block also considers information from pixels in neighboring blocks that are sufficiently close to the boundary. This is advantageous when filtering samples that are very close to the edge (e.g., the top-left sample of a block), because the sample now has samples to its left and above, which the filter can utilize to aid in filtering. Without this additional margin, the NN filter would not know how the image extends to the left and above, and would find it more difficult to improve such samples.

[0023] For example, when filtering a 128x128 sample block, eight samples from either side of the filter can be used in the input, making the input (128+8+8)x(128+8+8)=144x144, meaning there are 8 samples at each point. Choosing the appropriate margin size requires a trade-off: a small or non-existent margin reduces filtering quality; conversely, a large margin increases complexity.

[0024] In current existing techniques for neural network-based in-loop filters used for video compression / decompression, the output size is kept the same as the input size. For example, if the input can be as large as 144x144 samples, the output will also be as large as 144x144 samples. As a final step after filtering, the middle 128x128 samples are cropped from the output and used by the encoder / decoder. To maintain size, if a 3x3 kernel is used, one sample is padded to each side of the input (usually zeros) so that the output size of the layer is the same as the input size. In PyTorch, this is achieved by setting the input padding=(1,1) when calling the conv2d function. If a 5x5 kernel is used, padding=(2,2) must be used to keep the input and output sizes of the layer the same. Similarly, if a 3x1 kernel is used, padding=(1,0) will keep the input and output sizes of the layer the same.

[0025] Figure 16A and Figure 16B As shown, using padding=(1,1) will pad with one zero sample at each point outside the block. This will maintain the same dimensions after the 3x3 convolution.

[0026] In current existing techniques for neural network-based in-loop filters used for video compression / decompression, although the output size remains the same as the input size, the input size within the network sometimes changes by a factor of 2. For example, in Figure 12 In the middle, one of the layers is represented as .

[0027] This layer uses stride=(2,2) (e.g.) (As indicated in section 2) to reduce the output resolution to half its original value. If the input of this layer is 144x144, then the output of this layer will be 72x72. This layer can have any number of input and output channels, but the number of channels is typically increased when reducing spatial resolution. For example, the LOP2 model changes from 144x144x24 (24 channels) to 72x72x32 (32 channels).

[0028] Also in Figure 12 In this model, one layer is represented as pixel rearrangement. This layer doubles the resolution. For example, if the input to this layer is 72x72x4, the output will be 144x144x1, meaning the samples from four channels of size 72x72 are rearranged into one channel of size 144x144. When using pixel rearrangement, the number of output channels cannot be determined. If pixel rearrangement changes the resolution by a factor of 4 (in this case, a factor of 2 in width and a factor of 2 in height), the number of output channels must also be reduced to one-quarter of the original. Summary of the Invention

[0029] Several challenges exist. For example, high complexity is a major problem with current NN loop filter models or integrated NN filter models (hereinafter collectively referred to as "NN filters"). High complexity is a major obstacle to implementing and integrating these filters into everyday devices. Therefore, it is desirable to keep kMAC / kMULT, a measure of complexity, at a low level. As mentioned earlier, NN filters are currently applied to large pixel patches in images, and then the filter output is cropped to an appropriate size. Having a large patch size is equivalent to having a large margin, and since complexity is generally proportional to the number of samples in the input, a large margin will significantly increase kMAC / kMULT. For example, for a patch size of 128x128, a margin of 8 will increase the complexity by (128+8+8)x(128+8+8) / (128x128) = 1.26 times, that is, an increase of 26% compared to no margin; similarly, doubling the margin to 16 will increase the complexity by 56%.

[0030] Another problem with existing technologies is high energy consumption, which is directly related to the network complexity of NN filters. High energy consumption is a significant factor in both the training and inference phases.

[0031] Therefore, when using neural network (NN) filters to process images for video encoding and decoding, finding a proper balance between the performance and complexity / high energy consumption of the NN filter is crucial. It should be noted that the complexity of image processing is typically quantified by the number of multiply-accumulate operations per pixel, kMAC / pix (or simply kMAC), or the number of multiplications per pixel, kMULT / px (or simply kMULT), while the performance of the codec is defined by the improvement in compression efficiency, often simply referred to as "gain," and is usually measured as the Bjontegaard increment rate (BDR) relative to the anchor point. For example, a -1% BDR means that the same PSNR distortion can be achieved using 1% fewer bits.

[0032] To address the aforementioned issues, embodiments of this disclosure provide different ways to reduce the complexity and energy consumption of NN filters for image processing without significantly sacrificing performance.

[0033] More specifically, in one aspect of the embodiments of this disclosure, a method for processing an image is provided. The method includes selecting an input patch within an image. The input patch has a patch size (Yin, Xin), where Yin is the length of the input patch along a first direction and Xin is the length of the input patch along a second direction. The method further includes processing the input patch using a neural network (NN) filter to generate an output patch. The method also includes modifying the image at least based on the output patch. The NN filter includes convolutional layers C1 to C2 arranged sequentially. n Processing the input patch using NN filters includes: from the convolutional layer C... k Output the first tile with tile size (Y1, X1). k is less than or equal to n, Y1 is the length of the first tile along the first direction, X1 is the length of the first tile along the second direction, Y1 = a1 x Yin, X1 = b1 x Xin, each of Y1, X1, Yin, and Xin is an integer greater than or equal to 1, and both a1 and 1 / a1 are non-integers, and / or both b1 and 1 / b1 are non-integers.

[0034] In another aspect, a method for processing an image is provided. The method includes: selecting an input patch within an image, wherein the input patch has a patch size; processing the input patch using a neural network (NN) filter to generate an output patch; and modifying the image at least based on the output patch. The NN filter includes processing blocks arranged sequentially, including a processing block R, and the processing block R includes layers L1 to Ln arranged sequentially. Processing the input patch using the NN filter includes: inputting a first patch to the processing block R, wherein the input first patch is provided to layer L1 of the processing block R; cropping the input first patch to generate a cropped first patch, and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch. The size of the cropped first patch is smaller than the size of the first patch.

[0035] In various aspects, a computer program is provided that includes instructions that, when executed by processing circuitry, cause the processing circuitry to perform the methods of any of the above embodiments.

[0036] In various aspects, a carrier comprising the computer program of the above embodiments is provided. This carrier is one of electronic signals, optical signals, radio signals, and computer-readable storage media.

[0037] In various aspects, an apparatus for processing an image is provided. The apparatus is configured to select an input patch within an image. The input patch has a patch size (Yin, Xin), where Yin is the length of the input patch along a first direction, and Xin is the length of the input patch along a second direction. The apparatus is further configured to: process the input patch using a neural network (NN) filter to generate an output patch; and modify the image at least based on the output patch. The NN filter includes convolutional layers C1 to C2 arranged sequentially. n Processing the input patch using NN filters includes: from the convolutional layer C... k Output the first tile with tile size (Y1, X1). k is less than or equal to n, Y1 is the length of the first tile along the first direction, X1 is the length of the first tile along the second direction, Y1 = a1 x Yin, X1 = b1 x Xin, each of Y1, X1, Yin, and Xin is an integer greater than or equal to 1, and both a1 and 1 / a1 are non-integers, and / or both b1 and 1 / b1 are non-integers.

[0038] In various aspects, an apparatus for processing an image is provided. The apparatus is configured to: select an input patch within an image, wherein the input patch has a patch size; process the input patch using a neural network (NN) filter to generate an output patch; and modify the image at least based on the output patch. The NN filter includes processing blocks arranged sequentially, including a processing block R, and the processing block R includes layers L1 to Ln arranged sequentially. Processing the input patch using the NN filter includes: inputting a first patch to the processing block R, wherein the input first patch is provided to layer L1 of the processing block R; cropping the input first patch to generate a cropped first patch, and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch. The size of the cropped first patch is smaller than the size of the first patch.

[0039] In various aspects, an apparatus is provided that includes processing circuitry and a memory containing instructions executable by the processing circuitry, thereby enabling the apparatus to perform the methods of any of the above embodiments.

[0040] The embodiments of this disclosure can reduce the computational complexity of the NN model in the NN filter, for example, reducing the complexity of the low-operation-point (LOP) architecture NN loop filter by 0.5% while keeping the performance of the NN filter unchanged or at an acceptable level.

[0041] The embodiments can also accelerate the inference steps of the NN model by avoiding unnecessary computations, thereby reducing the total number of computations performed by the NN model. When the embodiments are used in an initial NN model before training, these embodiments can also shorten training time compared to existing NN models. This improvement stems from the reduction in the amount of computation required to achieve this. By avoiding unnecessary computations, the energy consumption of the NN model during the training phase can be reduced. It should be noted that the following embodiments can be directly applied to the inference phase after training using current prior art model architectures without affecting training, thus allowing for the reuse of previous training.

[0042] The following examples are applicable to most current NN-based in-loop filtering architectures. Attached Figure Description

[0043] The accompanying drawings, which are incorporated herein and form part of this specification, illustrate various embodiments.

[0044] Figure 1A A system according to some embodiments is shown.

[0045] Figure 1B and Figure 1C It shows Figure 1A An example of the system shown.

[0046] Figure 2A and Figure 2B This demonstrates how to reduce the size of the tiles input to a layer.

[0047] Figure 3A and Figure 3B This illustrates the effect of the input tile size on the output tile.

[0048] Figure 4 An example of a LOP2 neural network is shown.

[0049] Figure 5 The structure of an NN loop filter according to some embodiments is shown.

[0050] Figures 6A to 6B The text indicates "valid fill" (also known as "no fill").

[0051] Figures 6C to 6D This demonstrates how some computation can be saved by filtering samples on the boundary.

[0052] Figure 7 The structure of the backbone (also known as the trunk) block is shown.

[0053] Figure 8A The structure of the skeletal block is shown.

[0054] Figure 8B This demonstrates how the clipping operation for reconstruction can be performed.

[0055] Figure 9A and Figure 9B This demonstrates how virtual convolutional layers can be used to perform pruning operations.

[0056] Figure 10A The process according to some embodiments is shown.

[0057] Figure 10B The process according to some embodiments is shown.

[0058] Figure 11 An apparatus according to some embodiments is shown.

[0059] Figure 12 A unified neural network (NN) filter is shown.

[0060] Figure 13 The LOP architecture NN loop filter is shown.

[0061] Figure 14 This demonstrates how to use tiles to manipulate a portion of an image.

[0062] Figure 15 An example of an NN loop filter is shown.

[0063] Figure 16A and Figure 16B The concept of filling is illustrated.

[0064] Figure 17A A simplified view of an exemplary structure of an NN filter according to some embodiments is shown.

[0065] Figure 17B A separate clipping layer is shown according to some embodiments.

[0066] Figures 18A to 18B This illustrates the concept of applying convolution operations to a 3x3 input tile using a 2x2 kernel.

[0067] Figures 19A to 19B This demonstrates the concept of applying convolution operations to a 5x5 input tile using a 4x4 kernel without padding.

[0068] Figures 20A to 20B and Figure 21 This illustrates the concept of applying a convolution operation to a 5x5 input tile using a 4x4 kernel with (1,1) padding.

[0069] Figure 22 This illustrates the concept of applying a clipping operation in a feedforward path. Detailed Implementation

[0070] In this disclosure, the terms "neural network," "neural network model," and "NN model" are used interchangeably. Additionally, in this disclosure, "tile size" is used as a general term to refer to the size of data tiles in different arrangements (e.g., with different numbers of channels). Neural networks may contain layers or functions that rearrange data dimensions, such as pixel rearrangement; therefore, references to reducing tile size also include the different data arrangements before and after these layers or functions.

[0071] For example, suppose there is a first input with a width and height of 144 and 8 channels. This first input can be represented as 144x144x8. Further suppose there is a second input with a width and height of 144 but 4 channels. This second input can be represented as 144x144x4. Thus, the first and second inputs have the same tile size but different numbers of channels.

[0072] It should be noted that the 144x144 size is merely an example used to process a 128x128 sample region with 8 sample margins. The same neural network can be used with different sizes. If a 256x256 sample region is to be processed, then (256+8)x(256+8)=272x272 samples would be fed into the neural network. All tile sizes and kernel sizes mentioned below are provided for illustrative purposes only and do not limit the embodiments of this disclosure in any way.

[0073] Figure 1A A system 100 according to some embodiments is shown. System 100 includes a first entity 102, a second entity 104, and a network 110. The first entity 102 is configured to send a video stream (also referred to as a "video bitstream", "bitstream", or "encoded video") 106 to the second entity 104.

[0074] The first entity 102 can be any computing device (e.g., a network node such as a server) capable of encoding video using encoder 112 and transmitting the encoded video via network 110 to the second entity 104. The second entity 104 can be any computing device (e.g., a network node, a user device such as a mobile phone, laptop, tablet, vehicle, etc.) capable of receiving the encoded video and decoding it using decoder 114. Each of the first entity 102 and the second entity 104 can be a single physical entity or a combination of multiple physical entities. These multiple physical entities can be located in the same location or can be distributed in the cloud.

[0075] In some embodiments, such as Figure 1B As shown, the first entity 102 is a video streaming server 132, and the second entity 104 is a video streaming client, such as a user equipment (UE) 134. The UE 134 can be any of a desktop computer, laptop computer, tablet computer, mobile phone, head-mounted display, or any other computing device. The video streaming server 132 is capable of sending a video bitstream 136 (e.g., a movie) to the video streaming client 134. Upon receiving the video bitstream 136, the video streaming client 134 can decode the received video bitstream 136 to generate and display video for video streaming.

[0076] In other embodiments, such as Figure 1C As shown, the first entity 102 and the second entity 104 are the first UE 152 and the second UE 154. In one example, the first UE 152 is the initiator of the video conferencing session or the caller of the video chat, and the second UE 154 is the responder of the video conferencing session or the responder of the video chat. Figure 1CIn the illustrated embodiment, the first UE 152 can send a video bitstream 156 for video conferencing or video chat to the second UE 154. Upon receiving the video bitstream 156, the second UE 154 can decode the received video bitstream 156 to generate and display video for the video conferencing session or video chat.

[0077] As mentioned above, neural network (NN) filters can be used for encoding and decoding video. However, due to the complexity of the operations performed by the networks in NN filters, encoding and decoding video using NN filters may be inefficient. Therefore, a method is needed to reduce the computational complexity of NN filters while keeping their performance at an acceptable level.

[0078] Therefore, in one aspect of this disclosure, the size of the patch in each of one or more layers of the NN filter is reduced. By reducing the size of the patch in each of the one or more layers, the computational complexity of the NN filter can be reduced. Figure 2A and Figure 2B This demonstrates how to reduce the size of the tiles input to a layer.

[0079] exist Figure 2A In the NN filter shown, without considering the number of channels (i.e., C0 can be different from C), n When inputting a patch to the NN filter, the dimension (size) of the patch is the same as the dimension of the patch output from the NN filter. It should be noted that... Figure 2A In the NN filter shown, any change in patch size is factored by 2. For example, in Figure 2A In this process, layer 2 changes the tile size from 144x144 to 72x72, and layer n-2 changes the tile size back from 72x72 to 144x144. This is essentially a resolution change. In this disclosure, a change in tile size that is an integer multiple is called a resolution change. For example, since layer 2 changes the tile size from 144x144 (input) to 72x72 (output), and 144 / 72=2 is an integer, layer 2 is considered to have performed a resolution change. Similarly, since layer n-2 changes the tile size from 72x72 (input) to 144x144 (output), and 144 / 72=2 is an integer, layer n-2 is considered to have performed a resolution change.

[0080] As mentioned above, in Figure 2A In the NN filter shown, the input patch size is the same as the output patch size. In contrast, in... Figure 2BIn the NN filter shown, the size of the input patch (i.e., 144x144) differs from the size of the output patch (i.e., 136x136). This difference in patch size is achieved by the layers included in the NN filter. For example, Figure 2B Layer n-1 in the diagram changes the tile size by a non-integer multiple, i.e., from 140x140 to 138x138. Note that 144 / 138 = 1.04 and 138 / 144 = 0.958 are not integers. In another example, Figure 2B The layer n in the map changes the tile size in non-integer multiples, that is, from 138x138 to 136x136.

[0081] Reducing the tile size by a non-integer multiple in NN filters is to achieve a better trade-off between the computational complexity of NN filters and their performance (e.g., the quality of image processing performed by NN filters).

[0082] It should be noted that tile size reduction according to embodiments of this disclosure differs from simply rearranging samples or changing the resolution by an integer multiple. One purpose of reducing tile size according to embodiments of this disclosure is to remove sample values ​​that do not affect the useful portion of the NN filter output before they are processed by the layers of the NN filter, thereby preventing these sample values ​​from increasing the computational complexity of the NN filter.

[0083] Another difference between existing neural network (NN) filters and NN filters according to embodiments of this disclosure lies in how changes in the size of the input patch to the convolutional layer affect the size of the output patch. For example, in existing NN loop filters, the difference between the size of the input patch and the size of the output patch changes proportionally with the size of the input patch. In contrast, in embodiments of this disclosure, the patch size is reduced by a fixed amount regardless of the size of the input patch, so that the difference between the size of the input patch and the size of the output patch remains constant regardless of the size of the input patch. In other words, in the prior art, the input size (x) and the output size (y) have... The linear relationship between the input and output sizes is as follows. Therefore, in the prior art, if the input size is 144, the output size is 144; if the input size is reduced to 128, the output size will decrease to 128. The output size is a constant multiple of the input size, in this case, a multiple of 1, because 144 / 144 = 128 / 128 = 1. In contrast, in some embodiments of this disclosure, the input and output sizes do not follow this linear relationship. The relationship is non-linear. Therefore, changing the input size from 144 to 128 will cause the output size to change from 136 to 120. At this point, the input size and the output size are not constant multiples of each other: 144 / 136 is different from 128 / 120. Figure 3A and Figure 3BThis has been explained.

[0084] exist Figure 3B In the example, when an input patch of size 144x144 is processed through layers of an NN filter, the output size of the NN filter is reduced to a 136x136 output patch. Compared to the input size of 144x144, the reduced size of 136x136 is reduced by 8 in both the X and Y dimensions. Similarly, when an input patch of size 128x128 is processed through layers of an NN loop filter, the output size of the NN loop filter is reduced to a 120x120 output patch. As in the previous example, compared to the input size of 128x128, the reduced size of 120x120 is reduced by 8 in both the X and Y dimensions.

[0085] The idea is to reduce the tile size at the tail section.

[0086] Another concept of embodiments of this disclosure is to perform tile size reduction in the tail portion of the NN loop filter. Figure 4 This has been explained.

[0087] Figure 4 The LOP2 neural network implemented in NNVCv7 code is shown. In the prior art, the patch size of the input 401 of the NN filter is 144x144, the patch size of the output 405 of the last backbone layer of the NN filter is 72x72, and the patch size of each of the outputs 406, 407, 408, 409, and 410 of the subsequent layers CONV 1x3, CONV 3x1, CONV 1x1, PReLU, and CONV 3x3 is also 72x72. Finally, the pixel rearrangement layer receiving the output 410 with a patch size of 72x72 changes the resolution so that the patch size of the output 403 of the NN filter is 144x144. Subsequently, the middle portion of this 144x144 sample with a size of 128x128 is compared with... Figure 14 The 1403 samples are added together to obtain the final output.

[0088] In contrast, according to some embodiments of this disclosure, the tile size of output 405 of the last backbone block is 72x72, but the CONV 1x3 layer reduces the tile size to 72x70, i.e., the tile size of output 406 is 72x70, and the CONV 3x1 layer reduces the tile size to 70x70, i.e., the tile size of output 407 is 70x70. The subsequent two layers, CONV 1x1 and PReLU, do not change the tile size; therefore, the tile size of each of outputs 408 and 409 is 70x70. The last convolutional layer, CONV 3x3, further reduces the tile size to 68x68, i.e., the tile size of output 410 is 68x68. Finally, the pixel rearrangement layer increases the tile resolution by twice, from 68x68 to 136x136. Therefore, overall, the tile size changes in non-integer multiples.

[0089] In these embodiments, the tile size is changed several times before the last convolutional layer. This means that the last convolutional layer "CONV 3x3" only needs to perform 3x3x68x68xC. Y x4 multiplications. In contrast, in existing technologies, the same layer would require 3x3x72x72xC multiplications. Y x4 multiplications, resulting in a 12% saving. Similarly, similar savings can be achieved for earlier layers. For example, the CONV 1x1 layer in the above example operates on a tile size of 70x70, instead of operating on a tile size of 72x72 as in the prior art. Because the tile size has changed before this layer, the number of multiplications performed in this layer is significantly reduced. The changes.

[0090] This tile reduction aims to achieve a better trade-off between complexity and performance in neural network filters. More specifically, according to some embodiments of this disclosure, tile reduction is performed by pruning redundant computations on samples that do not contribute to the useful portion of the neural network filter output, thereby computing only on the sample data actually used in the neural network output. The resulting reduction in complexity can be used to improve the performance of the neural network filter, for example, by increasing the number of parameters or layers in the neural network filter.

[0091] General solutions

[0092] To reduce the computational complexity of the NN filter, in some embodiments of this disclosure, the NN model of the NN filter is modified to remove redundant computations, such as computations performed on boundary samples that will be pruned later. Redundant computations here are defined as computations that exist in inference but do not substantially affect the useful portion of the NN filter output, such as sample values ​​that will be used later within the pruned region.

[0093] Tile size can be reduced by removing samples from tile boundaries. In embodiments of this disclosure, the reduction in tile size is fixed and not proportional to the tile size itself. This means that even if the same neural network receives tiles of different sizes as input tiles, the reduction in tile size remains unaffected.

[0094] For example, suppose the tile size input to layer n is 144x144, and the tile size input to layer n+1 after layer n is 142x142. This size is 2 sample points smaller in both the X and Y dimensions. In this example, if the tile size input to layer n becomes 72x72, the tile size input to layer n+1 after layer n becomes 70x70, which is still 2 sample points smaller in both the X and Y dimensions.

[0095] More generally, suppose there are at least two layers, L1 and L2, in the neural network, where layer L2 is located after layer L1, the output patch size of layer L2 is smaller than the output patch size of layer L1, and the output patch size of layer L2 is not a divisor of the output patch size of layer L1. For example, see... Figure 2B The tile size change between the output of layer n-2 and the output of layer n-1. This size change does not simply stem from sample rearrangement, such as pixel rearrangement / pixel inversion operations. Furthermore, it is not due to a stride parameter not being 1. In these embodiments, the tile size reduction is not proportional to the tile size, but is fixed.

[0096] It should be noted that tile resizing can be achieved using clipping or other mechanisms. However, in this disclosure, the term "clipping" is used as a general term encompassing different types of resizing.

[0097] In some embodiments, the pruning step (e.g., pruning off a number of boundary pixels just before the output) is replaced by pruning redundant computations earlier in the preceding layers of the neural network. One example is replacing the final pruning step, which prunes a (n+2)x(n+2) patch size to an nxn patch size, with a pruning step performed just before the last 3x3 convolutional layer, which prunes the size of the patch input to the last 3x3 convolutional layer to (n+2)x(n+2). By pruning before the last convolutional step, the last convolution is applied to a smaller patch size, reducing computation while providing the same output as before. In the example above, it is assumed that no padding is applied to the last 3x3 convolution (or zero padding is applied), so the output patch size is unaffected by padding and only affected by the input patch size and kernel size.

[0098] Alternatively or additionally, a clipping step can be added to the feedforward path, passing the input tile forward to be added to the output tile. The added clipping step clips the input tile to the same size as the output tile before the addition step. Clipping steps can be added to each feedforward path, for example, one for luminance values ​​and another for chrominance values.

[0099] For example, Figure 13 The LOP architecture can be replaced with Figure 5 The architecture described above features a new approach where the final clipping boxes for both the luminance and chrominance branches are moved before the final convolution step, reducing computational cost in the last convolution step for both the luminance and chrominance channels. Additionally, two clipping boxes are added to the feedforward paths for both luminance and chrominance to match tile sizes before summing the values. This new architecture reduces computation while maintaining the same final output and therefore consistent output quality.

[0100] As an alternative to or supplement to providing a separate cropping layer, cropping effects can also be achieved by adjusting the padding mechanism (e.g., by changing the number of padding pixels). More specifically, convolutional layers with a kxk kernel size and zero padding will reduce the tile size on each side. Each pixel. For example, when k=5 and there is no padding, the size of the output patch of the convolutional layer will be smaller on each side. Two pixels. In this example, using two pixels of padding on each side will make the output tile size of the convolutional layer equal to the input tile size of the convolutional layer.

[0101] In some embodiments, the cropping step (e.g., cropping a number of boundary pixels just before the output) is replaced by a number of smaller cropping steps in the preceding layers. Each of these smaller cropping steps can be an actual cropping step introduced into the signal, or it can be the result of a convolutional layer with no padding or with minimal padding; for example, in the case of no padding, each 3x3 convolutional step crops one pixel from each side of the data tile. One way to implement this progressive cropping is to work backward from the last layer to calculate the required data tile size. In one example, if an output tile of size nxn pixels is required, and the current implementation provides a computed (n+8)x(n+8) data tile, which is then cropped to an nxn size, then the input to the last 3x3 convolutional layer requires at most (n+2)x(n+2) pixels, and the input to the 3x3 convolutional layers preceding the last convolutional layer requires at most (n+4)x(n+4) pixels, and so on. In another example, if the input patch size of the neural network is (n+16)x(n+16), the final cropping step of 16 pixels (8 pixels on each side) can be achieved simply by not padding the last eight convolutional layers, each with a 3x3 kernel size. Applying the proposed solution, the output patch size will directly be nxn. This avoids excessive computation on these boundary pixels without affecting important parts of the neural network output. Note that in PyTorch, no padding can be achieved by setting padding=(0,0).

[0102] In some embodiments, the pruning implementation of the luma (Y) branch can be the same as or different from the pruning implementation of the chroma (UV) branch. The final output sizes of luma and chroma can be the same, but pruning can be performed differently for the luma and chroma branches. In one example, the luma branch is pruned to a specific size in the last two layers, while the chroma branch is pruned in the last three layers, or the chroma branch is pruned to a different size in the last two layers.

[0103] Cutting by filling

[0104] As briefly mentioned above, one way to perform a clipping operation and thus avoid unnecessary computation is to choose a fill that is different from the "same" fill. In this disclosure, this is referred to as "clipping by fill." The "same" fill is a fill that makes the input tile size of the convolutional layer the same as the output tile size of the convolutional layer.

[0105] For example, suppose the last convolutional layer in an NN filter receives an input of size 144x144 samples and produces an output of 144x144 samples using a convolutional layer with a 3x3 kernel and padding=(1,1). Finally, this output is cropped by extracting the middle 128x128 samples. However, using padding=(1,1) can be wasteful because the topmost, bottommost, leftmost, and rightmost samples in the output will never be used due to the subsequent cropping to 128x128. Therefore, according to some embodiments of this disclosure, padding=(0,0) is used instead, i.e., no padding is performed in the last layer. This produces a smaller output of 142x142 samples, but the cropped result is the same since the subsequent cropping only uses the middle 128x128 samples. (0,0) padding is sometimes called “valid” padding because it only outputs values ​​at valid locations, where “valid” means that all parts of the filter are within the input. Another term for “valid” padding is “no padding” because the input is not expanded before filtering.

[0106] Figure 6A and Figure 6B The diagram illustrates how padding=(0,0) works. If the input size is n x n and the kernel size is 3 x 3, the output size will be (n-2) x (n-2). This saves computation because the number of multiplications required for padding=(1,1) will be 3 x 3 x 144 x 144 x Cin x Cout (where Cin is the number of input channels in the last layer and Cout is the number of output channels in the last layer), while with padding=(0,0), the number of multiplications is 3 x 3 x 142 x 142 x Cin x Cout, a reduction of 2.75%. It should be noted that with zero-padding blocks, the number of multiplications required to produce the output with padding=(1,1) can be reduced compared to the value of 3 x 3 x 144 x 144 x Cin x Cout reported here. This is because, as Figure 6C and Figure 6D As shown, this eliminates the need for some multiplications on boundary pixels. Figure 6C and Figure 6D As shown, for three of the nine positions in the 3x3 filter, the convolution result will be zero because the filter coefficients will be multiplied by zero. Therefore, in this case, only 2x3 = 6 multiplications are needed instead of 3x3 = 9. However, this is still much more computationally intensive than completely avoiding computation (compared to multiplying the coefficients by zero). For simplicity, this saving will be ignored in the remainder of this paper by removing the 3x3 terms on both sides, reducing the complexity from 144x144xCinxCout to 142x142xCinxCout.

[0107] Therefore, if a final output of size 128x128 is required, and the last layer is a 3x3 core, then inputting anything larger than 130x130 into the last layer would be wasteful. By strategically avoiding padding in the preceding layers, the input to the last layer can be set to a size of 130x130, and the output of that last layer can be set to a size of 128x128. For example, if the last eight layers are 3x3 conv2D layers, and the input of the first layer is 144x144, then by using padding=(0,0) in all eight layers, the output size of the first layer will be 142x142, the output size of the second layer will be 140x140, the output size of the third layer will be 138x138, and so on, until the output of the eighth layer (which is also the last layer), which will be 128x128. In this way, the complexity will be reduced from... Reduce to This reduces complexity by 12%. If there are more than eight layers in a neural network, padding=(1,1) can be used to maintain the same size until the last eight layers. For example, if the network consists of 100 3x3 convolutional layers, the first 100-8=92 layers can use padding=(1,1) to maintain the size of 144x144, but each of the last eight layers can use padding=(0,0) to achieve a size of 128x128 in the last layer.

[0108] Pruning by discarding data

[0109] Another way to perform clipping operations to avoid unnecessary computation is to discard some data before it is fed into the layer. In this disclosure, this is referred to as “correct clipping” to distinguish it from zero-padding (also known as clipping with padding). Therefore, in this disclosure, both “correct clipping” and “clipping with padding” are forms of clipping.

[0110] Suppose the input size of the last layer is 144x144, and the output size of the last layer is 144x144 before pruning and 128x128 after pruning. If the input of the last layer is pruned from 144x144 to 130x130, then the last layer will have an output of size 130x130 (this is achieved by using padding=(1,1)), after which the intermediate 128x128 samples can be pruned. This will produce the same result, but the number of multiplications in the last layer will be reduced from 144x144xCinxCout to 130x130xCinxCout, a reduction of 18%.

[0111] It should be noted that cropping by padding is generally preferred over proper cropping. This is because, in proper cropping, computational resources are used in the preceding layers to compute the signal (e.g., 144x144x8), but only some of these computed values ​​are subsequently discarded to obtain, for example, a signal of size 142x142x8. In contrast, cropping by padding does not compute these discarded values ​​from the outset. However, sometimes performing proper cropping may be preferred. For example, in some systems, there may be hardware support for performing a conv2D layer with padding=(1,1), but a conv2D layer with padding=(0,0) might be performed in software. In this case, it is preferable to compute all values ​​in hardware and then discard some of them.

[0112] Pruning by filling and discarding data

[0113] Another way to perform clipping to avoid unnecessary calculations is by clipping through padding, discarding some data before it's fed into the layer. Here's an example of this combination:

[0114] First, proper clipping can be applied to reduce the tile size from 144x144 to 130x130. Then, clipping by padding (using padding=(0,0)) can be performed to directly obtain the desired output size of 128x128. This further reduces the complexity in the last layer from 144x144xCinxCout to 128x128xCinxCout, a reduction of 21%.

[0115] It should be noted that whether "correct cropping" or "cropping by discarding" is used, cropping is different from the resolution change defined earlier in this article. This means that the tile sizes before and after the layer do not satisfy either input size / output size = integer or output size / input size = integer.

[0116] In some cases, the input tile size is not square. For example, if the tile is filtered at the edges of the image, the tile size may not be 144x144, but 144x64. In this case, a conv2d with kernel=3x3 and padding=(0,0) will produce a 142x62 output.

[0117] In some cases, the kernel size is not a square like 3x3 or 5x5, but can be, for example, 3x1. Typically, a 3x3 kernel can be approximated with two layers: a 3x1 layer and a 1x3 layer. In the 3x1 case, padding=(1,0) is used for "same" padding; in the 1x3 case, padding=(0,1) is used for "same" padding. However, padding=(0,0) is still used for "valid" padding. Therefore, in one respect, if the input of the penultimate layer is 130x130, and the last two layers are 3x1 and 1x3, padding=(0,0) can be used in the penultimate layer to transform it from 130x130 to 128x130, and padding=(0,0) can be used in the last layer to transform it from 128x130 to 128x128.

[0118] Reduce tile size in advance

[0119] In some embodiments, the tile size can be reduced in earlier layers of the neural network architecture, and then the tile size can be kept constant, for example, by using padding over the remaining layers of the NN architecture. For example, the neural network architecture consists of a head portion, a backbone portion, and a tail portion, each portion having one or more convolutional layers, and according to some embodiments, the tile size can be reduced within the head portion and before the backbone portion. In this example, the tile size is then kept constant throughout the backbone and tail portions. In another example, the reduction of the tile size begins in the backbone portion, and once the tile size reaches the output tile size, a padding mechanism, for example, is used to keep the tile size constant.

[0120] In some embodiments, instead of performing tile size reduction in every consecutive convolutional layer, the tile size is kept constant in some of these convolutional layers. In one example, the tile size is reduced in convolutional layer L, then maintained in convolutional layers L+1 and L+2, and then reduced again in convolutional layer L+3.

[0121] More specifically, considering the complexity savings that smaller sizes can bring, it may be advantageous to switch to smaller sizes earlier in the network. For example, suppose the NN filter consists of 100 3x3 conv2D layers, and for simplicity, assume Cin = Cout = 1. Further assume the complexity of the first 92 layers is (144x144), that is, The complexity of the last eight layers is In these embodiments, padding=(0,0) can be applied to the first eight layers. This means that at the output of the eighth layer, the size will be reduced to 128x128. At this point, the remaining 92 layers can have padding=(1,1) applied to maintain the size at (128x128). The cost of the first eight layers remains 145968, but the cost of the last 92 layers will be significantly reduced. By performing the pruning in advance, the total complexity will be reduced from 1907712+145968 to 145968+1507328, saving an additional 20%.

[0122] It should be noted that these embodiments may require new training of the network in the NN filter, or the same trained weights of the original architecture may be reused.

[0123] Use the reduction in complexity to improve performance.

[0124] In some embodiments, the complexity reduction achieved in the above embodiments is returned to the NN loop filter by increasing the input patch size of the neural network in the NN loop filter. In these embodiments, the size of the input patch of the neural network is increased so that the neural network receives patches of a larger size compared to the original input patch size. Subsequently, in order for the neural network to provide the same output patch size, the patch size is adjusted through different layers of the neural network, for example, by appropriate padding or introducing intermediate pruning steps. The increased input patch size may increase the computational complexity of the NN loop filter, but it can also improve the performance of the filter.

[0125] In these embodiments, the size of the input patch can be increased to a degree that keeps the total complexity of the NN filter operations constant. In other words, the increased complexity caused by larger patch sizes in earlier layers can be compensated by the reduced complexity caused by smaller patch sizes in later layers. It should be noted that these embodiments may require retraining the neural network or may reuse the same trained weights from the original architecture.

[0126] In some embodiments, the complexity reduction achieved in the above embodiments is returned to the NN loop filter by increasing the network parameters of the neural network in the NN loop filter. Examples of increasing network parameters include, but are not limited to, adding more layers to the network, increasing the kernel size in one or more layers, etc. It should be noted that in these embodiments, it may be necessary to retrain the neural network in the NN loop filter through a complete retraining or fine-tuning step.

[0127] For example, suppose the original NN loop filter network has a complexity of 17 kMAC / pixel. Further suppose that after applying the concepts described above, the complexity is reduced to 16 kMAC / pixel, and the performance results are the same or similar. If 17 kMAC / pixel is an acceptable complexity, then there is an additional budget of 1 kMAC / pixel that can be used to make the network deeper, wider, or with more branches, or otherwise make the network larger. With proper training, such network deepening or widening will produce a BD rate saving that is superior to the original network at the same complexity. Thus, the gain in terms of complexity savings can be traded for a quality gain, measured in BD rate.

[0128] Increase the kernel size and tile size of the intermediate layers

[0129] In some embodiments, the kernel size of one or more intermediate convolutional layers can be increased while employing a larger padding size. This increases the tile size. Subsequently, in later layers or after several layers, progressive padding can be applied to crop the larger tiles, thereby reducing the size to the original tile size.

[0130] Reduce tile size within the residual block.

[0131] As mentioned above, in Figure 4 In the NN loop filter shown, when the patch size of the output 405 of the last backbone block is 72x72, the output 405 can be processed by using CONV 1x3 layer, CONV 3x1 layer and CONV 3x3 layer to reduce the patch size of the output 410 of CONV 3x3 layer to 68x68. Therefore, the output 403 of the pixel rearrangement layer can have a patch size of 136x136.

[0132] However, in some scenarios, further reducing the tile size of output 403 to a tile size of 128x128 might be better. This is because, as shown in Figure 16, the output of the NN loop filter is generated by correcting the sample data in part 1503 of the input image 1501 using output 403, and 128x128 is used as the size of that part 1503.

[0133] To further reduce the tile size of output 403, the output 405 of the last skeleton block should be set to a smaller value. For example, if the output 405 of the last skeleton block is 68x68, the output of subsequent layers will look like this:

[0134]

[0135] As shown in the table above, if the output 405 of the last backbone block is 68x68, then the patch size of the final residual output 403 of the NN loop filter is 128x128, which is exactly the size required to correct the sample data in part 1503 of the input image 1501. To reduce the patch size of the output 405 of the last backbone block to 68x68, in some embodiments of this disclosure, one or more backbone blocks included in the NN loop filter are modified.

[0136] Figure 7 This illustrates how a backbone block in an NN loop filter typically operates. This backbone block is a residual block. This means that the input 701 to the backbone block is forwarded not only to the first layer of the block (“CONV 1x1 CxC1”) but also to the adder 710.

[0137] The following table shows the size of the input 701 and outputs 702 to 709 of the backbone block.

[0138]

[0139] As shown above, the size of input 701 is the same as that of output 707, both being 72x72xC. This allows the two to be added together using adder 710, resulting in output 709, which also has a size of 72x72xC. It should be noted that although the number of channels changes, the tile size remains the same (72x72) throughout the entire backbone block.

[0140] Figure 8A This illustrates how the backbone block in an NN loop filter operates according to some embodiments of the present disclosure. Figure 8A In the first layer, the size of input 801 is 70x70xC, not 72x72xC. Subsequently, the first three layers ("CONV 1x1 CxC1", "PReLU", and "CONV 1x1 C1xC") do not change size, so the size of output 804 of the "CONV 1x1 C1xC" layer is 70x70xC.

[0141] However, the next layer "sCONV 1x3 CxC" 21 "By setting the fill to (0,0) to perform the clipping operation, the output 805 is 70x68xC (tile size 70x68). This is consistent with..." Figure 7 The examples shown are different. Figure 7 In the middle, due to "sCONV 1x3 CxC 21 The layer uses a kernel with padding=(0,1) to perform convolution operations, so the input and output sizes of this layer are the same. In contrast, here the padding is set to (0,0), so "sCONV 1x3 CxC21 "The convolution operation performed by the kernel in the layer reduces the tile size."

[0142] Similarly, the subsequent layer "sCONV 3x1 C 21 "xC" performs another clipping operation by setting the fill to (0,0), resulting in an output size of 68x68xC (tile size 68x68) for 806. The last layer, "sCONV 1x1 CxC", does not change the tile size, so its output size for 807 will be 68x68xC.

[0143] To add output 807 to input 801, in some embodiments, a cropping operation 811 is performed on input 801. This is because the size of input 801 (i.e., 70x70xC) does not match the size of output 807 (i.e., 68x68xC). Cropping operation 811 removes boundary samples (i.e., the topmost, bottommost, leftmost, and rightmost samples) from each channel, resulting in an output 808 of size 68x68xC. Adder 810 can be used to add output 808 to output 807, producing an output 809 of the same size 68x68xC. The tile sizes of inputs / outputs 801 to 809 are shown below.

[0144]

[0145] As described above, in order for the last backbone block to generate an output 809 with a tile size of 68x68, the input 801 of this block should be 70x70. This means that the output 809 of the penultimate backbone block should also have a size of 70x70. Assuming the input of the penultimate backbone block has a size of 72x72xC (i.e., a tile size of 72x72), the penultimate backbone block can reduce its tile size in the same way that the last backbone block reduced its tile size from 70x70 to 68x68. The tile sizes of the inputs / outputs 801 to 809 of the penultimate backbone block are shown below.

[0146]

[0147] Therefore, with the NN loop filter comprising 12 backbone blocks and the first backbone block having an input size of 72x72xC, the first 10 backbone blocks can operate without any pruning. The last two backbone blocks can undergo pruning to reduce the patch size from 72x72 to 68x68. Subsequently, the output of the last backbone block can be provided as input 405 to the remaining layers of the NN loop filter, thereby generating an output 403 with a size of 128x128x1 as described above.

[0148] In summary, in the above embodiments, not all backbone blocks of the NN loop filter are architecturally identical, because some of these blocks (e.g., the last two backbone blocks) can change their tile size, while the remaining blocks (e.g., the first ten backbone blocks) can remain unchanged in tile size. This contrasts with the case where all backbone blocks have similar architectures and therefore do not change their tile size.

[0149] like Figure 8A As shown, the output of the last convolutional layer of the backbone block is combined with the input of the backbone block to generate the output of the backbone block. In order to combine the output of the last convolutional layer with the input of the backbone block, the output size of the last convolutional layer should be the same as the input size of the backbone block.

[0150] However, as mentioned above, when the tile size of the last two backbone blocks is reduced, the output size of the last convolutional layer will differ from the input size of the backbone blocks. To address this issue, in some embodiments of this disclosure, a pruning operation 811 is performed on the input 801. The purpose of the pruning operation 811 is to reduce the size of the input 801 so that the output size of the last convolutional layer is the same as the input size of the backbone blocks.

[0151] There are different ways to perform the trimming operation 811.

[0152] In one example, the pruning operation 811 can be performed simply by not forwarding all input samples 801 to the adder 810. More specifically, as Figure 8B As shown, the cropping operation 811 can be performed in the following way: instead of forwarding the input sample 801 corresponding to region 852 to adder 810, only the input sample 801 corresponding to region 854 is forwarded.

[0153] In another example, due to some limitations in the software, it may not be possible to perform the aforementioned "selective" forwarding of input samples, so a virtual convolutional layer can be used to simulate the pruning operation. Figure 9A and Figure 9B This has been explained.

[0154] exist Figure 9A In this process, an input 901 of size 70x70xC is fed through a 3x3 convolutional layer with C groups, and this input will undergo a pruning operation 911. Having C groups means that each channel will perform convolution independently of each other. Here, setting the kernel weights to zero except for the middle position, setting the bias to 0, and setting the padding to (0,0) will result in an output 908 of size 68x68xC (or, if the input 901 is nxnxC, the output 908 will be (n-2)x(n-2)xC).

[0155] Another way to simulate this cropping operation is to use two one-dimensional convolutional layers, such as... Figure 9B As shown. In Figure 9B First, an input 951 of size (n+2)x(n+2)xC is fed into a 1x3 convolutional layer with C groups and padding = (0,0). By setting the kernel weights of this convolutional layer to [0, 1, 0] and the bias term to 0, a pruning operation can be simulated, resulting in an output 970 of size (n+2)xnxC. Then, this output is fed into a 3x1 convolutional layer with C groups and padding = (0,0). By setting the kernel weights of this convolutional layer to [0, 1, 0] and the bias term to 0, a pruning operation can be simulated, resulting in an output 970 of size (n+2)xnxC.

[0156]

[0157] Setting the bias term to 0 can simulate another clipping operation, so that the output 958 of clipping operation 961 has a size of n x n x C.

[0158] In some cases, the performance of the filter can be further improved by using convolution to perform bypass link pruning, while allowing weights outside the center to be non-zero and allowing intermediate weights to be different from 1.

[0159] Figure 10A An image processing procedure 1000 according to some embodiments is illustrated. Procedure 1000 may begin at step s1002. Step s1002 includes selecting an input patch within an image. The input patch has a patch size (Yin, Xin), where Yin is the length of the input patch along a first direction, and Xin is the length of the input patch along a second direction. Step s1004 includes processing the input patch using a neural network (NN) filter to generate an output patch. Step s1006 includes modifying the image at least based on the output patch. The NN filter includes convolutional layers C1 to C2 arranged sequentially. n Processing the input patch using NN filters includes: from the convolutional layer C... k Output the first tile with tile size (Y1, X1). k is less than or equal to n, Y1 is the length of the first tile along the first direction, and X1 is the length of the first tile along the second direction. Y1 = a1 x Yin, X1 = b1 x Xin, each of Y1, X1, Yin, and Xin is an integer greater than or equal to 1, and both a1 and 1 / a1 are non-integers, and / or both b1 and 1 / b1 are non-integers.

[0160] In some embodiments, processing the input patch using an NN filter further includes: obtaining a second patch; and providing the second patch to the convolutional layer C. kThe first tile is generated by applying a convolution operation to the second tile using a convolutional layer Ck. The second tile has a tile size (Y2, X2), where Y2 is the length of the second tile along a first direction, X2 is the length of the second tile along a second direction, Y1 = a2 x Y2, X1 = b2 x X2, each of Y2 and X2 is an integer greater than or equal to 1, and both a2 and 1 / a2 are non-integers, and / or both b2 and 1 / b2 are non-integers.

[0161] In some embodiments, a convolution operation is applied to the second tile using a kernel of size (Yk, Xk) and padding of size (Yp, Xp), where floor(Yk / 2)>Yp and / or floor(Xk / 2)>Xp, each of Yk and Xk is an integer greater than or equal to 1, and each of Yp and Xp is an integer greater than or equal to 0.

[0162] In some embodiments, processing the input patch using an NN filter further includes: obtaining a second patch; cropping the second patch by removing samples at the boundaries of the second patch to generate a cropped second patch; and providing the cropped second patch to the convolutional layer C. k ; and by applying convolution operations to the cropped second tile, using a convolutional layer C k Generate the first tile. The second tile has a tile size (Y2, X2), where Y2 is the length of the second tile along the first direction, X2 is the length of the second tile along the second direction, Y1 = a2 x Y2, X1 = b2 x X2, each of Y2 and X2 is an integer greater than or equal to 1, and both a2 and 1 / a2 are non-integers, and / or both b2 and 1 / b2 are non-integers.

[0163] In some embodiments, the cropped second patch has a size (Y2', X2'), where Y2' is the length of the cropped second patch along a first direction and X2' is the length of the cropped second patch along a second direction. A convolution operation is applied to the cropped second patch using a kernel of size (Yk, Xk) and padding of size (Yp, Xp). , Each of Yk and Xk is an integer greater than or equal to 1, and each of Yp and Xp is an integer greater than or equal to 0.

[0164] In some embodiments, Y2'=Y1 and X2'=X1.

[0165] In some embodiments, Y2'>Y1 and X2'>X1.

[0166] In some embodiments, k is less than n, and obtaining the second patch includes: obtaining the third patch; providing the third patch to the convolutional layer C. k-1 This generates the second tile.

[0167] In some embodiments, Y2-Y1 are fixed in advance and are independent of the value of Y2, and X2-X1 are fixed in advance and are independent of the value of X2.

[0168] In some embodiments, the convolutional layers include a first group of one or more convolutional layers and a second group of one or more convolutional layers, and processing the input patches using NN filters includes: cropping the patches to be input to each convolutional layer in the first group by removing samples at patch boundaries, and providing the cropped patches to each convolutional layer in the first group; and / or cropping the patches to each convolutional layer in the second group by using convolution operations. The first group and the second group may be different or the same.

[0169] In some embodiments, the NN filter includes a head portion, a backbone portion, and a tail portion arranged sequentially, and the first set of convolutional layers and the second set of convolutional layers are included only in the head portion, only in the backbone portion, or only in the tail portion.

[0170] In some embodiments, the NN filter includes a head portion, a backbone portion, and a tail portion, and a first set of convolutional layers and a second set of convolutional layers are included in two or more of the head portion, the backbone portion, and the tail portion.

[0171] In some embodiments, the NN filter includes one or more processing blocks arranged sequentially, and the size of the patch input to at least one of the one or more processing blocks is different from the size of the patch output from the at least one of the one or more processing blocks.

[0172] In some embodiments, the one or more processing blocks include a processing block R, which includes a first group of one or more convolutional layers and / or a second group of one or more convolutional layers; processing the input patch using an NN filter includes: obtaining the input patch P. R-In The input image block P is processed by one or more convolutional layers in the first group and one or more convolutional layers in the second group. R-In The process is performed to generate the output tile P. R-Out Output block P R-Out Provided to the adder; for the input block P R-In The input tile P is cropped to generate a cropped input tile. R-In-Cropped ; the cropped input tile P R-In-Cropped Forward to the adder; and use the adder to output tile P R-Outwith the cropped input patch P R-In-Cropped to generate a combined patch. An output patch is generated based on the combined patch.

[0173] In some embodiments, for input patch P R-In cropping is performed by not forwarding all samples included in input patch P R-In to an adder.

[0174] In some embodiments, for input patch P R-In cropping is performed by using at least one convolutional layer, the at least one convolutional layer has a kernel of size (Yk, Xk) and does not have any padding, or has padding of size (Yp, Xp), wherein Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

[0175] In some embodiments, all weights of the kernel are zero except for the intermediate weights, and a bias term of the at least one convolutional layer is zero.

[0176] In some embodiments, input patch P R-In has a size (Y R-In , X R-In ), Y R-In is the length of input patch P R-In along a first direction, X R-In is the length of input patch P R-In along a second direction, output patch P R-Out has a size (Y R-Out , X R-Out ), Y R-Out is the length of the output patch PR-Out along the first direction, X R-Out is the length of output patch P R-Out along the second direction, Y R-Out = a3 x Y R-In , X R-Out = b3 x X R-In , Y R-In , X R-In , Y R-Out and X R-Out each is an integer greater than or equal to 1, and both a3 and 1 / a3 are non-integers, and / or both b1 and 1 / b1 are non-integers.

[0177] In some embodiments, the NN filter is an in-loop filter in an image or video encoder or decoder.

[0178] In some embodiments, the one or more processing blocks are residual blocks included in an image or video encoder or decoder.

[0179] In some embodiments, the output patch includes reconstructed samples.

[0180] Figure 10B An image processing process 1050 is shown. Process 1050 may start at step s1502. Step s1502 comprises selecting an input patch within an image, wherein the input patch has a patch size. Step s1504 comprises processing the input patch using a neural network (NN) filter, thereby generating an output patch. Step s1506 comprises modifying the image based at least on the output patch. The NN filter comprises processing blocks arranged in sequence, the processing blocks including a processing block R, which includes layers L1 to Ln arranged in sequence; processing the input patch using the NN filter comprises: inputting a first patch into the processing block R, wherein the input first patch is provided to the layer L1 of the processing block R; cropping the input first patch, thereby generating a cropped first patch, and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from the layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch. The size of the cropped first patch is smaller than the size of the first patch.

[0181] In some embodiments, cropping the input first patch is performed by not forwarding all samples included in the first patch input to the processing block R to the adder.

[0182] In some embodiments, cropping the input first patch is performed by using at least one convolutional layer, the at least one convolutional layer having a kernel of size (Yk, Xk) and a padding of size (Yp, Xp), and Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

[0183] In some embodiments, all weights of the kernel except an intermediate weight are zero, and the bias term of the at least one convolutional layer is zero.

[0184] Figure 11 is a block diagram of an apparatus 1100 for implementing an encoder 112 and / or a decoder 114 in accordance with some embodiments. When the apparatus 1100 implements a decoder, the apparatus 1100 may be referred to as "decoding apparatus 1100"; and when the apparatus 1100 implements an encoder, the apparatus 1100 may be referred to as "encoding apparatus 1100". As Figure 11As shown, device 1100 may include: a processing circuit (PC) 1102, which may include one or more processors (P) 1155 (e.g., a general-purpose microprocessor and / or one or more other processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), which may coexist in a single housing or a single data center, or may be geographically distributed (i.e., device 1100 may be a distributed computing device); at least one network interface 1148, which includes a transmitter (Tx) 1145 and a receiver (Rx) 1147 for enabling device 1100 to send data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) that is (directly or indirectly) connected to the network interface 1148 (e.g., the network interface 1148 may be wirelessly connected to the network 110, in which case the network interface 1148 is connected to an antenna arrangement); and a storage unit (also referred to as a “data storage system”) 1108, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1102 includes a programmable processor, a computer program product (CPP) 1141 may be provided. CPP 1141 includes a computer-readable medium (CRM) 1142 storing a computer program (CP) 1143 including computer-readable instructions (CRI) 1144. CRM 1142 may be a non-transitory computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a memory device (e.g., random access memory, flash memory), etc. In some embodiments, the CRI 1144 of the computer program 1143 is configured such that, when executed by PC 1102, the CRI causes device 1100 to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, device 1100 may be configured to perform the steps described herein without code. That is, for example, PC 1102 may consist only of one or more ASICs. Therefore, the features of the embodiments described herein may be implemented in hardware and / or software.

[0185] The core concepts of some embodiments of this disclosure can be summarized as follows.

[0186] When processing images, neural networks (“NN”) can be used. The NN used for image processing can be called an NN filter. In one example, an NN filter is an NN loop filter included in an image / video encoder and can be used to encode images / videos. In another example, an NN filter is an NN loop filter included in an image / video decoder and can be used to decode images / videos. Figure 17A A simplified view of an exemplary structure of the NN filter 1700 is shown.

[0187] like Figure 17A As shown, the NN filter 1700 includes multiple convolutional layers 1702, 1704, 1706, 1708, ... arranged sequentially. Following the order of the convolutional layers, the output of one convolutional layer is used as input to another convolutional layer. For example, in... Figure 17A In this configuration, the output of layer 1702 is provided as input to layer 1704, the output of layer 1704 is provided as input to layer 1706, and so on. It should be noted that one or more non-convolutional layers may exist between any two adjacent convolutional layers. An example of such a non-convolutional layer is the Parametric Corrected Linear Unit (PReLU). It should also be noted that not all convolutional layers included in the NN filter 1700 are arranged sequentially. For example, the NN filter 1700 may include one or more convolutional layers (e.g., convolutional layer 1710) connected in parallel with other convolutional layers (e.g., layer 1702).

[0188] In the NN filter 1700, a patch can be provided as input to each of a plurality of convolutional layers, and each convolutional layer can perform a convolution operation on the input patch using a kernel. As is known in the art, a convolution operation includes performing mathematical operations such as multiplication and addition on the sample values ​​of the patch input to each convolutional layer using the value of the kernel.

[0189] For example, Figure 18A The input patch 1802 to the convolutional layer and the kernel 1804 of the convolutional layer are shown. Figure 18A As shown, input patch 1802 includes sample values ​​P11, P12, P13, P21, P22, P23, P31, P32, and P33, and kernel 1804 includes values ​​K11, K12, K21, and K22. For example, as... Figure 18B As shown in Figure 18E, a convolution operation is performed on the input patch 1802 by moving the kernel 1804 and applying the kernel value to the sample values. For example, by... Figure 18BAs shown in Figure 18C, kernel 1804 is applied to the top-left four pixels of input patch 1802 to obtain sample value O11 of output patch 1806; as shown in Figure 18D, kernel 1804 is applied to the top-right four pixels to obtain sample value O12 of output patch 1806; as shown in Figure 18D, kernel 1804 is applied to the bottom-left four pixels to obtain sample value O21 of output patch 1806; and as shown in Figure 18E, kernel 1804 is applied to the bottom-right four pixels to obtain sample value O22 of output patch 1806. A bias value B can also be associated with the kernel; this bias value can be added after convolution. (If there are more output channels, there can be more than one bias value.) The sample values ​​of output patch 1806 can be calculated as follows: O11 = P11 x K11 + P12 x K12 + P21 x K21 + P22 x K22 + B, O12 = P12 x K11 + P13 x K12 + P22 x K21 + P23 x K22 + B, O21 = P21 x K11 + P22 x K12 + P31 x K21 + P32 x K22 + B, and O22 = P22 x K11 + P23 x K12 + P32 x K21 + P33 x K22 + B.

[0190] The computational complexity of a convolutional layer is proportional to the number of multiplications and additions required to perform the convolution operation, and the number of multiplications and additions required to perform the convolution operation is proportional to the size of the input patch 1802. Therefore, to reduce the computational complexity of the convolutional layers included in the NN filter, the size of the input patch to be provided to the convolutional layer can be reduced. This size reduction can be called a pruning operation.

[0191] For convolutional layers, there are two different ways to perform the clipping operation. The first way is to remove sample values ​​at the boundaries of the tiles to be input into the convolutional layer; the second way is to perform a convolution operation with no padding or minimal padding at the previous convolutional layer and / or the current convolutional layer (i.e., the convolutional layer to which the tiles are input).

[0192] For example, suppose there is an input patch 1712 of size (N x M) to be fed into convolutional layer 1708, where N and M can be the same or different. In order to reduce the complexity of the convolution operation performed by convolutional layer 1708, the size of the input patch 1712 needs to be reduced before it is fed into convolutional layer 1708.

[0193] The first way to reduce the size of input tile 1712 is: (e.g.) Figure 17BAs shown, a separate clipping layer 1720 is provided between the previous convolutional layer 1706 and convolutional layer 1708. This separate clipping layer 1720 is not a convolutional layer. The clipping layer 1720 is configured to receive input patch 1712, remove sample values ​​at the boundaries of input patch 1712, thereby generating a clipped input patch 1722 of size (N-C1 x M-C2), where (C1>0 and C2>=0), (C1>=0 and C2>0), or (C1>0 and C2>0), where C1 corresponds to the number of pixels clipped along a first direction (e.g., the y-direction), and C2 corresponds to the number of pixels clipped along a second direction (e.g., the x-direction).

[0194] The second approach is to reduce the size of the input patch 1712 by performing a no-padding (i.e., padding to (0,0)) clipping operation at the previous convolutional layer 1704. For example, as Figure 18A As shown, when a convolution operation is performed on a tile without padding, the output of the convolution operation is a tile smaller than the size of the tile after the convolution operation. For example, in Figure 18A In this context, the input patch 1802 is 3x3 in size, while the output patch 1806 is 2x2 in size. Therefore, by performing convolution operations at any one or more of the previous convolutional layers 1702, 1704, and 1706, the size of the input patch 1712 to be provided to the convolutional layer 1708 can be reduced.

[0195] In some embodiments, even when filling is required, cropping can be achieved by performing a convolution operation on the tiles. Figure 19A , Figure 19B , Figure 20A and Figure 20B This has been explained.

[0196] Figure 19A This demonstrates performing a convolution operation on a 5x5 input tile using a 4x4 kernel without padding. Figure 19B As shown, the convolution operation is performed by moving the kernel until there are no sample values ​​in the 5x5 input patch that have not been processed by the kernel. This convolution operation will produce a 2x2 output patch (corresponding to the number of kernel moves). Here, the size difference between the input patch and the output patch is (5-2, 5-2) = (3,3).

[0197] Figure 20A The execution and Figure 19A The same convolution operation is shown, but with padding (1,1). The areas where padding is applied correspond to squares marked "P". Note that "P" does not indicate any specific value, but only that padding has been applied to that area. Figure 20A As shown, applying a fill of (1,1) to a 5x5 input tile will produce a 7x7 tile. Figure 20B and Figure 21 As shown, convolution is performed by moving the kernel until there are no unprocessed sample values ​​in the 7x7 tile. This convolution operation produces a 4x4 output tile (corresponding to the number of kernel moves). Here, the size difference between the input and output tiles is (5-4, 5-4) = (1,1). This shows that even with non-zero padding, in the case of floor(Ky / 2) > Py and / or floor(Kx / 2) > Px, the clipping operation can be performed solely by performing the convolution operation.

[0198] In some embodiments, one or more convolutional layers (e.g., 1702 to 1708) included in the NN filter 1700 are included in a processing block of the NN filter 1700, which includes a feedforward path. For example, Figure 22 A simplified structure of a processing block (e.g., a residual block) 2200 is shown. Figure 22 As shown, based on the last convolutional layer included in processing block 2100 (e.g., Figure 22 The output 2204 of the last convolutional layer 1708 is summed with the input 2206 of the processing block 2200 to determine the output 2202 of the processing block 2200. It should be noted that one or more additional layers (for processing the output 2204 of the last convolutional layer 1708) may exist between the last convolutional layer 1708 and the adder 2212. However, for ease of illustration, this example will assume that the output 2204 of the last convolutional layer 1708 is provided to the adder 2212.

[0199] When the plot to be input to any one or more convolutional layers 1702 to 1708 is cropped as described above, the size of the input 2206 of processing block 2200 will be different from the size of the output 2204 of the last convolutional layer 1708. More specifically, the size of the output 2204 of the last convolutional layer 1708 will be smaller than the size of the input 2206 of processing block 200. Therefore, according to some embodiments, an additional cropping operation can be performed on the input 2206 of processing block 2200 in the feedforward path, such that the size of the cropped input 2206' forwarded to adder 2212 is the same as the size of the cropped output 2204. As described above, the cropping of the input can be performed by a convolutional layer or a separate cropping layer.

[0200] Summary of Implementation Examples

[0201] Compared to existing technologies, in the proposed solution, the size of the output patch of the last layer of the NN loop filter differs from the size of the input patch of the NN loop filter.

[0202] In some embodiments of the proposed solution, the size difference between the input tile size and the output tile size is fixed and therefore independent of the input tile size and its variations.

[0203] In some embodiments of the proposed solution, the input size and output size are not integer multiples of each other; that is, the input size / output size is not an integer, and the output size / input size is not an integer either.

[0204] In the proposed method, there is at least one resizing step between two layers of the neural network, wherein the resizing step may include cropping or zero padding.

[0205] In some embodiments of the proposed solution, there is no further pruning step after the last layer of the NN loop filter.

[0206] In some embodiments of the proposed solution, all computations in the neural network affect the values ​​of useful pixels in the output patch of the last layer of the neural network.

[0207] In some embodiments of the proposed method, the output tile size has been achieved in an intermediate layer of the NN loop filter. Here, an intermediate layer refers to any layer that is neither the first nor the last layer.

[0208] In some embodiments of the proposed method, tile size reduction is performed in multiple steps. For example, a 10-pixel tile size reduction is performed across five different convolutional layers, each with a 3x3 kernel and zero-pixel padding at the boundaries. In this example, each convolutional layer will reduce the tile size by 2 pixels in the X dimension and 2 pixels in the Y dimension.

[0209] In some embodiments of the proposed method, the neural network model is trained using equal input and output tile sizes, as described in the prior art, but the inference steps are then performed using the proposed model with an output tile size smaller than the input tile size. In other words, a model following the proposed method can reuse the training of prior art models.

[0210] In another embodiment, the complexity of the proposed method is reduced and returned to the model by increasing model parameters (e.g., making the model deeper and wider, or increasing the kernel size, increasing the tile size, etc.), thereby improving the model's performance.

[0211] In some embodiments of the proposed solution, the input tile size is large, and the output tile size is smaller than the input and output tile sizes of prior art NN filters.

[0212] In some embodiments of the proposed solution, there is a pruning step in the feedforward path of the NN architecture.

[0213] Additional summary of the embodiments

[0214] A1. A method (1000) for processing an image, the method comprising: selecting (s1002) an input patch within an image, wherein the input patch has a patch size (Yin, Xin), and Yin is the length of the input patch along a first direction, and Xin is the length of the input patch along a second direction; processing the input patch using a neural network (NN) filter (s1004) to generate an output patch; and modifying the image at least based on the output patch (s1006), wherein the NN filter includes convolutional layers C1 to C2 arranged sequentially. n (That is, the convolutional layers are arranged sequentially, starting from convolutional layer C1 and ending at convolutional layer C2.) n This ensures that for i=1 to n, the convolutional layer C i In this sequence, it immediately follows the convolutional layer C i-1 Then, and immediately following the convolutional layer C i+1 Previously); processing the input patch using NN filters includes: from convolutional layer C k Output the first tile with tile size (Y1, X1), where k is less than or equal to n, and Y1 is the length of the first tile along the first direction, X1 is the length of the first tile along the second direction, Y1 = a1 x Yin, X1 = b1 x Xin, each of Y1, X1, Yin and Xin is an integer greater than or equal to 1, and both a1 and 1 / a1 are non-integers, and / or both b1 and 1 / b1 are non-integers.

[0215] Clipping is performed at the convolutional layer.

[0216] A2. The method according to embodiment A1, wherein processing the input patch using an NN filter further includes: obtaining a second patch; and providing the second patch to the convolutional layer C. k ; and by applying a convolution operation to the second tile, a first tile is generated using a convolutional layer Ck; the second tile has a tile size (Y2, X2), where Y2 is the length of the second tile along a first direction, X2 is the length of the second tile along a second direction, Y1 = a2 x Y2, X1 = b2 x X2, each of Y2 and X2 is an integer greater than or equal to 1, and both a2 and 1 / a2 are non-integers, and / or both b2 and 1 / b2 are non-integers.

[0217] Clipping is performed at the convolutional layer.

[0218] A3. The method according to embodiment A2, wherein a convolution operation is applied to the second tile using a kernel of size (Yk, Xk) and a padding of size (Yp, Xp), floor(Yk / 2)>Yp and / or floor(Xk / 2)>Xp, each of Yk and Xk is an integer greater than or equal to 1, and each of Yp and Xp is an integer greater than or equal to 0.

[0219] Pruning is performed before the convolutional layer.

[0220] A4. The method according to embodiment A1, wherein processing the input patch using an NN filter further includes: obtaining a second patch; cropping the second patch by removing samples at the boundaries of the second patch to generate a cropped second patch; and providing the cropped second patch to the convolutional layer C. k ; and by applying convolution operations to the cropped second tile, using a convolutional layer C k Generate the first tile; the second tile has a tile size (Y2, X2), where Y2 is the length of the second tile along the first direction, X2 is the length of the second tile along the second direction, Y1 = a2 x Y2, X1 = b2 x X2, each of Y2 and X2 is an integer greater than or equal to 1, and both a2 and 1 / a2 are non-integers, and / or both b2 and 1 / b2 are non-integers.

[0221] Pruning can be performed before the convolutional layer but not at the convolutional layer itself, or pruning can be performed before the convolutional layer and additional pruning can be performed at the convolutional layer.

[0222] A5. The method according to embodiment A4, wherein the cropped second patch has a size (Y2', X2'), where Y2' is the length of the cropped second patch along a first direction, X2' is the length of the cropped second patch along a second direction, and a convolution operation is applied to the cropped second patch using a kernel of size (Yk, Xk) and padding of size (Yp, Xp). , Each of Yk and Xk is an integer greater than or equal to 1, and each of Yp and Xp is an integer greater than or equal to 0.

[0223] Pruning is performed before the convolutional layer but not at the convolutional layer itself.

[0224] A6. The method according to embodiment A5, wherein Y2'=Y1 and X2'=X1.

[0225] Clipping is performed before the convolutional layer and additional clipping is performed at the convolutional layer.

[0226] A7. The method according to embodiment A5, wherein Y2'>Y1 and X2'>X1.

[0227] A8a. The method according to any one of embodiments A2 to A7, wherein k is less than n, and obtaining the second patch includes: obtaining a third patch; providing the third patch to the convolutional layer C. k-1 This generates the second tile.

[0228] A8b. The method according to embodiment A8a, wherein the third patch is composed of convolutional layer C k-2 Generate, k>2.

[0229] A9. The method according to any one of embodiments A2 to A8b, wherein Y2-Y1 are predetermined and independent of the value of Y2, and / or X2-X1 are predetermined and independent of the value of X2.

[0230] A10. The method according to any one of embodiments A1 to A9, wherein the convolutional layer includes a first group of one or more convolutional layers and a second group of one or more convolutional layers, and processing the input patch using an NN filter includes: cropping the patch to be input to each convolutional layer included in the first group by removing samples at the patch boundaries, and providing the cropped patch to each convolutional layer included in the first group; and / or cropping the patch to each convolutional layer included in the second group by using a convolution operation; and the first group and the second group are different or the same.

[0231] A11. The method according to embodiment A10, wherein the NN filter includes a head portion, a backbone portion and a tail portion arranged sequentially, and the first set of convolutional layers and the second set of convolutional layers are included only in the head portion, only in the backbone portion, or only in the tail portion.

[0232] A12. The method according to embodiment A10, wherein the NN filter includes a head portion, a backbone portion, and a tail portion, and a first set of convolutional layers and a second set of convolutional layers are included in two or more portions of the head portion, the backbone portion, and the tail portion.

[0233] A13. The method according to any one of embodiments A10 to A12, wherein the NN filter includes one or more processing blocks arranged sequentially, and the size of the tile input to at least one of the one or more processing blocks is different from the size of the tile output from the at least one of the one or more processing blocks.

[0234] A14. The method of embodiment A13, wherein the one or more processing blocks comprise processing block R, the processing block R comprises a first set of one or more convolution layers and / or a second set of one or more convolution layers; processing an input patch with an NN filter comprises: obtaining input patch P R-In ; processing the input patch P by the first set of one or more convolution layers and the second set comprised in processing block R R-In to generate output patch P R-Out ; providing the output patch P R-Out to an adder; cropping the input patch P R-In to generate a cropped input patch P R-In-Cropped ; forwarding the cropped input patch P R-In-Cropped to the adder; and using the adder to add the output patch P R-Out and the cropped input patch P R-In-Cropped to generate a combined patch; the output patch is generated based on the combined patch.

[0235] A14a. The method of embodiment A14, wherein the cropping of input patch P R-In is performed by not forwarding all samples comprised in input patch P R-In to the adder.

[0236] A14b. The method of embodiment A14, wherein the cropping of input patch P R-In is performed by using at least one convolution layer, the at least one convolution layer has a kernel of size (Yk, Xk) and does not have any padding, or has padding of size (Yp, Xp), wherein Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

[0237] A14c. The method of embodiment A14b, wherein all weights of the kernel are zero except for an intermediate weight, and the bias term of the at least one convolution layer is zero.

[0238] A15. The method of any one of embodiments A14 to A14c, wherein the input patch P R-In has a size (Y R-In , X R-In ), Y R-In is the length of the input patch P R-In along a first direction, X R-In is the length of the input patch P R-In along a second direction, the output patch P R-Out has a size (Y R-Out , X R-Out ), YR-Out It is the output block P R-Out The length along the first direction, X R-Out It is the output block P R-Out The length along the second direction, Y R-Out =a3 x Y R-In X R-Out =b3 x X R-In Y R-In X R-In Y R-Out and X R-Out Each of the elements in the set is an integer greater than or equal to 1, and both a3 and 1 / a3 are non-integers, and / or both b1 and 1 / b1 are non-integers.

[0239] A16. The method according to any one of embodiments A1 to A15, wherein the NN filter is a loop filter in an image or video encoder or decoder.

[0240] A17. The method according to embodiment A16, wherein when embodiment A16 is subordinate to any one of embodiments A13 to A15, wherein the one or more processing blocks are residual blocks included in an image or video encoder or decoder.

[0241] A18. The method according to embodiment A16 or A17, wherein the output tiles contain reconstructed samples.

[0242] B1. A method (1050) for processing an image, the method comprising: selecting (s1052) an input patch within an image, wherein the input patch has a patch size; processing the input patch using a neural network (NN) filter (s1054) to generate an output patch; and modifying the image at least based on the output patch (s1056), wherein the NN filter includes processing blocks arranged sequentially, the processing blocks including a processing block R, the processing block R including layers L1 to Ln arranged sequentially; processing the input patch using the NN filter includes: inputting a first patch to the processing block R, wherein the input first patch is provided to layer L1 of the processing block R; cropping the input first patch to generate a cropped first patch, and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch; wherein the size of the cropped first patch is smaller than the size of the first patch.

[0243] B2. The method according to embodiment B1, wherein cropping the input first tile is performed by not forwarding all samples included in the first tile input to the processing block R to the adder.

[0244] B3. The method of embodiment B1, wherein cropping the input first patch is performed by using at least one convolution layer, the at least one convolution layer has a kernel with a size of (Yk, Xk) and a padding with a size of (Yp, Xp), and Yp<floor(Yk / 2) and / or Xp<floor(Xk / 2).

[0245] B4. The method of embodiment B3, wherein all weights of the kernel are zero except for intermediate weights, and the bias term of the at least one convolution layer is zero.

[0246] C1. A computer program (1100), comprising instructions (1144) which, when executed by a processing circuit (1102), cause the processing circuit to perform the method according to any one of embodiments A1 to B4.

[0247] C2. A carrier comprising the computer program of embodiment C1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal and a computer readable storage medium.

[0248] D1. An apparatus (1100) for processing an image, the apparatus being configured to: select (s1002) an input patch within the image, wherein the input patch has a patch size (Yin, Xin), and Yin is the length of the input patch along a first direction, and Xin is the length of the input patch along a second direction; process (s1004) the input patch using a neural network NN filter, thereby generating an output patch; and modify the image at least based on the output patch (s1006), wherein the NN filter comprises convolution layers C1 to C n ; processing the input patch with the NN filter comprises: from convolution layer C k outputting a first patch having a patch size (Y1, X1), wherein k is less than or equal to n, Y1 is the length of the first patch along the first direction, X1 is the length of the first patch along the second direction, Y1 = a1 x Yin, X1 = b1 x Xin, each of Y1, X1, Yin and Xin is an integer greater than or equal to 1, and both a1 and 1 / a1 are non-integers, and / or both b1 and 1 / b1 are non-integers.

[0249] D2. The apparatus according to embodiment D1, wherein the apparatus is configured to perform the method according to any one of embodiments A2 to A18.

[0250] E1. An image processing apparatus (1100) configured to: select (s1052) an input patch within an image, wherein the input patch has a patch size; process the input patch using a neural network (NN) filter (s1054) to generate an output patch; and modify the image at least based on the output patch (s1056), wherein the NN filter includes processing blocks arranged sequentially, including a processing block R, the processing block R including layers L1 to Ln arranged sequentially; processing the input patch using the NN filter includes: inputting a first patch to the processing block R, wherein the input first patch is provided to layer L1 of the processing block R; cropping the input first patch to generate a cropped first patch, and inputting the cropped first patch to an adder via a bypass path; outputting a second patch from layer Ln; providing the second patch to the adder; and combining the second patch with the cropped first patch; wherein the size of the cropped first patch is smaller than the size of the first patch.

[0251] E2. The apparatus according to embodiment E1, wherein the apparatus is configured to perform the method according to any one of embodiments B2 to B4.

[0252] F1. An apparatus (1100) comprising: a processing circuit (1102); and a memory (1141) containing instructions executable by the processing circuit, thereby enabling the apparatus to perform the method according to any one of embodiments A1 to B4.

[0253] Although various embodiments have been described herein, it should be understood that these embodiments are given by way of example only and not by way of limitation. Therefore, the breadth and scope of this disclosure should not be limited to any of the exemplary embodiments described above. Furthermore, unless otherwise stated herein or the context clearly contradicts it, this disclosure covers any combination of the foregoing elements in all possible variations.

[0254] As used herein, sending a message "to" or "towards" a intended recipient encompasses sending the message directly to the intended recipient or sending the message indirectly to the intended recipient (i.e., relaying the message from the source node to the intended recipient using one or more other nodes). Similarly, as used herein, receiving a message "from" a sender encompasses receiving the message directly from the sender or receiving the message indirectly from the sender (i.e., relaying the message from the sender to the receiving node using one or more nodes). Furthermore, as used herein, "a" means "at least one" or "one or more".

[0255] Furthermore, although the process described above and shown in the accompanying figures is presented as a sequence of steps, this is for illustrative purposes only. Therefore, it is conceivable that some steps can be added, some steps can be omitted, the order of the steps can be rearranged, and some steps can be performed in parallel.

[0256] Reference List

[0257] [1] P. Cerwall (Executive Editor) et al., Ericsson Mobility Report, November 2019 (available at www.ericsson.com / en / mobility-report).

[0258] [2] EE1-1.6: Combined Test of EE1-1.2 and EE1-1.4, Yue Li, Kai Zhang, Li Zhang, Hongtao Wang, Jianle Chen, Kevin Reuze, AnandMeher Kotra, Marta Karczewicz, JVET-X0066, October 2021.

[0259] [3] EE1-1.2: Test on Deep In-Loop Filter with Adaptive Parameter Selection and Residual Scaling, Yue Li, Kai Zhang, Li Zhang, Hongtao Wang, Kevin Reuze, Anand Meher Kotra, MartaKarczewicz, JVET-Y0143, January 2022.

[0260] [4] BoG report on NN-filter design unification, E. Alshina and F. Galpin, JVER-AD0380, April 2023.

[0261] [5] BoG Report: Neural Network Technology, A. Segall, JVET-T0130, October 2020.

Claims

1. A method for processing an image (1000), the method comprising: Select (s1002) an input patch within the image, wherein the input patch has a patch size (Yin, Xin), where Yin is the length of the input patch along a first direction and Xin is the length of the input patch along a second direction; The input patch is processed using a neural network (NN) filter (s1004) to generate an output patch; and The image is modified at least based on the output tiles (s1006), wherein, The NN filter includes convolutional layers C1 to C2 arranged sequentially. n , Processing the input patch using the NN filter includes: From convolutional layer C k Output a first tile with tile size (Y1, X1), where k is less than or equal to n, Y1 is the length of the first tile along the first direction, and X1 is the length of the first tile along the second direction. Y1 = a1 x Yin, X1 = b1 x Xin, Each of Y1, X1, Yin, and Xin is an integer greater than or equal to 1, and Both a1 and 1 / a1 are non-integers, and / or both b1 and 1 / b1 are non-integers.

2. The method according to claim 1, wherein, Processing the input patch using the NN filter further includes: Obtain the second tile; The second image patch is provided to the convolutional layer C. k ;as well as By applying a convolution operation to the second image patch, using a convolutional layer C k Generate the first tile. The second tile has a tile size of (Y2, X2). Y2 is the length of the second patch along the first direction, and X2 is the length of the second patch along the second direction. Y1 = a2 x Y2, X1 = b2 x X2, Each of Y2 and X2 is an integer greater than or equal to 1, and Both a2 and 1 / a2 are non-integers, and / or both b2 and 1 / b2 are non-integers.

3. The method according to claim 2, wherein, The convolution operation is applied to the second tile using a kernel of size (Yk, Xk) and padding of size (Yp, Xp). floor(Yk / 2)>Yp, and / or floor(Xk / 2)>Xp, Each of Yk and Xk is an integer greater than or equal to 1, and Each of Yp and Xp is an integer greater than or equal to 0.

4. The method according to claim 1, wherein, Processing the input patch using the NN filter further includes: Obtain the second tile; The second patch is cropped by removing samples from its boundaries, thereby generating a cropped second patch. The cropped second image patch is provided to the convolutional layer C. k ;as well as By applying a convolution operation to the cropped second image patch, using a convolutional layer C... k Generate the first tile. The second tile has a tile size of (Y2, X2). Y2 is the length of the second patch along the first direction, and X2 is the length of the second patch along the second direction. Y1 = a2 x Y2, X1 = b2 x X2, Each of Y2 and X2 is an integer greater than or equal to 1, and Both a2 and 1 / a2 are non-integers, and / or both b2 and 1 / b2 are non-integers.

5. The method according to claim 4, wherein, The cropped second tile has a size of (Y2', X2'). Y2' is the length of the cropped second patch along the first direction, and X2' is the length of the cropped second patch along the second direction. The convolution operation is applied to the cropped second tile using a kernel of size (Yk, Xk) and padding of size (Yp, Xp). ， ， Each of Yk and Xk is an integer greater than or equal to 1, and Each of Yp and Xp is an integer greater than or equal to 0.

6. The method according to claim 5, wherein, Y2'=Y1 and X2'=X1.

7. The method according to claim 5, wherein, Y2'>Y1 and X2'>X1.

8. The method according to any one of claims 2 to 7, wherein, k is less than n, and Obtaining the second tile includes: Obtain the third tile; and The third image block is provided to the convolutional layer C. k-1 k>1, thus generating the second tile.

9. The method according to claim 8, wherein, The third image patch is composed of convolutional layer C. k-2 Generate, k>2.

10. The method according to any one of claims 2 to 9, wherein, Y2-Y1 is fixed in advance and is independent of the value of Y2, and / or X2-X1 is fixed in advance and is independent of the value of X2.

11. The method according to any one of claims 1 to 10, wherein, The convolutional layer includes a first group of one or more convolutional layers and a second group of one or more convolutional layers, and Processing the input patch using the NN filter includes: By removing samples at the tile boundaries, the tiles to be input to each convolutional layer in the first group are cropped, and the cropped tiles are provided to each convolutional layer in the first group; and / or By using convolution operations, the patches input to each convolutional layer included in the second group are cropped, and The first group and the second group are either different or the same.

12. The method according to claim 11, wherein, The NN filter includes a head portion, a backbone portion, and a tail portion arranged sequentially, and The first set of convolutional layers and the second set of convolutional layers are included only in the head portion, only in the backbone portion, or only in the tail portion.

13. The method according to claim 11, wherein, The NN filter includes a head portion, a backbone portion, and a tail portion, and The first set of convolutional layers and the second set of convolutional layers are included in two or more of the head portion, the backbone portion, and the tail portion.

14. The method according to any one of claims 11 to 13, wherein, The NN filter includes one or more processing blocks arranged sequentially, and The size of the tile input to at least one of the one or more processing blocks is different from the size of the tile output from at least one of the one or more processing blocks.

15. The method according to claim 14, wherein, The one or more processing blocks include processing block R, which includes one or more convolutional layers from the first group and / or one or more convolutional layers from the second group. Processing the input patch using the NN filter includes: Obtain input tile P R-In ; The input image block P is processed by the first group of one or more convolutional layers and the second group of one or more convolutional layers included in the processing block R. R-In The process is performed to generate the output tile P. R-Out ; The output block P R-Out Provided to the adder; For the input block P R-In The input tile P is cropped to generate a cropped input tile. R-In-Cropped ; The cropped input image P R-In-Cropped Forwarded to the adder; and Using the adder, the output block P R-Out With the cropped input tile P R-In-Cropped Add them together to generate a composite tile, and The output tile is generated based on the combined tile.

16. The method according to claim 15, wherein, For the input block P R-In Cropping is performed by not cropping the input tile P. R-In All samples included in the process are forwarded to the adder for execution.

17. The method according to claim 15, wherein, For the input block P R-In cropping is performed by using at least one convolutional layer, the at least one convolutional layer has a kernel with a size of (Yk, Xk) and does not have any padding, or has a padding with a size of (Yp, Xp), wherein Yp < floor(Yk / 2) and / or Xp < floor(Xk / 2).

18. The method according to claim 17, wherein, Except for the intermediate weights, all weights of the kernel are zero, and The bias term of at least one convolutional layer is zero.

19. The method according to any one of claims 15 to 18, wherein, The input block P R-In Having size (Y) R-In , X R-In ), Y R-In The input block P R-In The length along the first direction, X R-In The input block P R-In Length along the second direction, The output block P R-Out Having size (Y) R-Out , X R-Out ), Y R-Out The output block P R-Out The length along the first direction, X R-Out The output block P R-Out Length along the second direction, AND R-Out =a3 x Y R-In , X R-Out =b3 x X R-In , Y R-In X R-In Y R-Out and X R-Out Each of the elements is an integer greater than or equal to 1, and Both a3 and 1 / a3 are non-integers, and / or both b1 and 1 / b1 are non-integers.

20. The method according to any one of claims 1 to 19, wherein, The NN filter is a loop filter in an image or video encoder or decoder.

21. The method of claim 20, when claim 20 is dependent on any one of claims 14-19, wherein the one or more processing blocks are residual blocks comprised in said image or video encoder or decoder.

22. The method according to claim 20 or 21, wherein, The output block comprises reconstructed samples.

23. A method (1050) of processing an image, the method comprising: selecting (s1052) an input block within the image, wherein the input block has a block size; processing the input block (s1054) using a neural network NN filter, thereby generating an output block; and modifying the image (s1056) based at least on the output block, wherein, the NN filter comprises processing blocks arranged in sequence, the processing blocks include a processing block R, the processing block R comprises layers L1 to Ln arranged in sequence, processing the input block using the NN filter comprises: inputting a first block into the processing block R, wherein the input first block is provided to the layer L1 of the processing block R; cropping the input first block, thereby generating a cropped first block, and inputting the cropped first block via a bypass path to an adder; outputting a second block from the layer Ln; providing the second block to the adder; and combining the second block with the cropped first block, and the size of the cropped first block is smaller than the size of the first block.

24. The method according to claim 23, wherein, cropping the input first block is performed by not forwarding all samples comprised in the first block input to the processing block R to the adder.

25. The method of claim 23, wherein, cropping the input first block is performed by using at least one convolutional layer, said at least one convolutional layer has a kernel with a size of (Yk, Xk) and has a padding with a size of (Yp, Xp), and Yp<floor(Yk / 2) and / or Xp<floor(Xk / 2).

26. The method of claim 25, wherein, all weights of the kernel are zero except for the intermediate weight, and a bias term of said at least one convolutional layer is zero.

27. A computer program (1100), comprising instructions (1144), which when executed by a processing circuit (1102), cause the processing circuit to perform the method according to any one of claims 1 to 26.

28. A carrier comprising the computer program according to claim 27, wherein, the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

29. An apparatus (1100) for processing an image, the apparatus is configured to: Select the input patch within the image described in (s1002), wherein, the input block has a block size (Yin, Xin), Yin is the length of the input block along a first direction, and Xin is the length of the input block along a second direction; The input patch is processed using a neural network (NN) filter (s1004) to generate an output patch. as well as The image is modified at least based on the output tiles (s1006), wherein, The NN filter includes convolutional layers C1 to C2 arranged sequentially. n , Processing the input patch using the NN filter includes: From convolutional layer C k Output a first tile with tile size (Y1, X1), where k is less than or equal to n, Y1 is the length of the first tile along the first direction, and X1 is the length of the first tile along the second direction. Y1 = a1 x Yin, X1 = b1 x Xin, Each of Y1, X1, Yin, and Xin is an integer greater than or equal to 1, and Both a1 and 1 / a1 are non-integers, and / or both b1 and 1 / b1 are non-integers.

30. The apparatus according to claim 29, wherein, The apparatus is configured to perform the method according to any one of claims 2 to 22.

31. An image processing apparatus (1100), the apparatus being configured to: Select the input patch within the image described in (s1052), wherein, The input tile has a tile size; The input patch is processed using a neural network NN filter (s1054) to generate an output patch; as well as The image is modified at least based on the output tiles (s1056), wherein, The NN filter comprises processing blocks arranged sequentially. The processing block includes processing block R. The processing block R includes layers L1 to Ln arranged sequentially. Processing the input patch using the NN filter includes: The first image block is input into the processing block R, wherein the input first image block is provided to layer L1 of the processing block R; The input first tile is cropped to generate a cropped first tile, and the cropped first tile is input into the adder via a bypass path; Output the second tile from layer Ln; Provide the second block to the adder; and Combine the second image block with the cropped first image block, and The size of the cropped first patch is smaller than the size of the first patch.

32. The apparatus according to claim 31, wherein, The apparatus is configured to perform the method according to any one of claims 24 to 26.

33. An apparatus (1100) comprising: Memory; as well as Processing circuit, in which, The apparatus is configured to perform the method according to any one of claims 1 to 26.

Images