Neural network for image processing

Abstract

A method for image processing. The method includes obtaining a first input patch comprising a first set of one or more processed samples within a first region of an image. The method also includes processing the first input patch using a first part of an NN-based filter to produce a first set of NN outputs. The method also includes obtaining a first output patch based on a determined type of the first input patch, the first output patch comprising a second set of one or more processed samples within the first region of the image.

Description

NEURAL NETWORK FOR IMAGE PROCESSINGTECHNICAL FIELD

[0001] This disclosure relates to image processing, such as, video encoding and video decoding, and, more specifically, to image processing systems and methods employing one or more artificial neural networks (ANNs) (or “neural networks (NNs)” for short).BACKGROUND

[0002] Video is the dominant form of data traffic in today’s networks and is projected to still increase its share. One way to reduce the data traffic from video is compression. Here the source video is encoded to a bitstream, which then can be stored and transmitted to end users. Using a decoder, the end user can extract the video data and display it on a screen. However, because the encoder may not know what kind of device the encoded bitstream is going to be sent to, it has to compress the video to a predetermined format, such as a standardized. This way, all devices that support the chosen standard can decode the video. Compression can be lossless, i.e., the decoded video will be identical to the source given to the encoder, or lossy, where a certain degradation of content is accepted. Using lossy compression allows for significantly lower bit rates, i.e., the compression ratio can be much higher. This is because reproducing image noise perfectly can make lossless compression quite expensive.

[0003] A video (a.k.a., “video sequence”) contains a sequence of pictures. A color space commonly used in videos is YCbCr, where Y is the luma (brightness) component and Cb and Cr are the chroma components. Sometimes the Cb and Cr components are called U and V. Other color spaces are also used, such as ICtCp, IPT, constant-luminance YCbCr, RGB, YCoCg etc., and the present invention is applicable also in these cases.

[0004] VVC and HEVC

[0005] Versatile Video Coding (VVC) and its predecessor High Efficiency Video Coding (HEVC) are block-based video codecs standardized and developed jointly by ITU-T and MPEG. The codecs utilize both temporal and spatial prediction. VVC and HEVC are similar in many aspects. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using uni-directional (P) or bi-directional inter (B) prediction on a block level from previously decoded reference pictures. In theencoder, the difference between the original pixel data and the predicted pixel data, referred to as the residual, is transformed into the frequency domain, quantized and then entropy coded before transmitted together with necessary prediction parameters such as prediction mode and motion vectors, also entropy coded. The decoder performs entropy decoding, inverse quantization and inverse transformation to obtain the residual, and then adds the residual to an intra or inter prediction to reconstruct a picture. The VVC version 1 specification was published as Rec. ITU-T H.266 | ISO / IEC 23090-3, “Versatile Video Coding”, in 2020.

[0006] Blocks and Units

[0007] In many video coding standards, such as HEVC and VVC, each component is split into blocks and the coded video bitstream consists of a series of coded blocks. A block is a two-dimensional array of samples. It is common in video coding that the picture is split into units that cover a specific area of the picture. Each unit consists of all blocks from all components that make up that specific area and each block belongs fully to one unit. The macroblock in H.264 and the Coding unit (CU) in HEVC and VVC are examples of units. A “transform block” is a block that a transform used in coding is applied to. A “prediction block” is a block that a single prediction mode is applied to.

[0008] Exploration Experiment on Neural Network based Video Coding (NNVC)

[0009] At the 20th JVET meeting it was decided to set up an exploration experiment (EE) on neural network-based (NN-based) video coding. The exploration experiment continued at the subsequent JVET meetings 21 through 31 with many tests: NN-based in-loop filtering, NN- based post filtering, NN-based super resolution and NN-based intra prediction.

[0010] VVC contains three in-loop filters that are not based on neural networks: A deblocking filter, a sample adaptive offset (SAG) filter, and an adaptive loop filter (ALF). The deblocking filter is used to remove block artifacts by smoothening discontinuities in horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS can have values 0, 1, and 2, where a larger value indicates a stronger filtering. The output of the deblocking filter is further processed by SAG, and the output of SAG is then processed by ALF.

[0011] The output of the ALF filter can then be put into the decoded picture buffer (DPB), which contains decoded pictures that may be used for prediction of subsequently encoded (ordecoded) pictures. Since the deblocking filter, SAO filter and ALF filter in this way influence the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loop filters. This means that changes done by the loop filter may influence not only the current picture but future pictures.

[0012] It is possible for a decoder to further filter the picture in the DPB, but not store the filtered output in the DPB. In contrast to loop filters, such a filter is not influencing future predictions and is therefore classified as a post-processing filter (or “post-filter” or “postfilter”). Postfiltering is generally optional for decoders and thereby not required to be performed for decoder implementations to conform to a standard specification.

[0013] The contributions JVET-X0066 and JVET-Y0143 are two successive contributions that describe NN-based in-loop filtering. Both contributions use the same NN models for filtering. The NN-based in-loop filter is placed before SAO and ALF and the samples before the deblocking filter are used as input to the filter. The output of the NN-based in-loop filter is mixed with the output of the deblocking filter and forwarded as the input to SAO.

[0014] The purpose of using the NN-based filter is to improve the quality of the reconstructed samples. Here it is helpful that the NN model is non-linear. While the deblocking filter, SAO, and ALF all contain non-linear elements such as conditions, and are thus not strictly linear, all three of them are based on linear filters. In contrast, a sufficiently large NN model can in principle leam any non-linear mapping and is therefore capable of representing a wider class of functions compared to deblocking, SAO and ALF. In JVET- X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters. In a refined version of that work presented in the contribution JVET-AB0052, only two models are used: One for luma samples and another model for chroma samples.

[0015] JVET-AD0380 proposed a new unified design for the NN-based in-loop filtering, which captures the benefits of previous NN structures. The unified filter has only one NN model, to filter luma and chroma samples and intra and inter pictures. There are six inputs, the reconstructed samples of luma and chroma before deblocking (‘rec’), the prediction samples of luma and chroma (‘pred’), the BS information of luma and chroma (‘bs’), the quantization parameter for a sequence (‘QPbase’), the quantization parameter for each slice CQPslice’), as well as information on whether a particular sample was intra-predicted, unipredicted or bi-predicted (TPB’). These inputs first go through a convolutional layer (3x3 or1x1) and a parametric rectified linear unit (PReLU) layer separately, then they are concatenated and fused together with a 1x1 convolutional layer.

[0016] The NN-based in-loop filters presented in JVET-X0066, JVET-AB0053, JVET- AB0052 and JVET-AD0380 increase the compression efficiency of the codec substantially, i.e., they lower the bit rate substantially without lowering the objective quality as measured by MSE-based PSNR. Increases in compression efficiency, often referred to simply as “gain”, are often measured as the Bjontegaard-delta rate (BDR) against an anchor. As an example, a BDR of -1% means that the same PSNR distortion can be reached with 1% bitrate saving on average. As reported in JVET-AF0041, for the random access (RA) configuration, the BDR for the luma component (Y) is -10.27%, and for the all-intra (Al) configuration, the BDR for the luma component is -7.86%.

[0017] The complexity of NN models used for compression are often measured by MAC / pixel (multiply-accumulate operations per pixel). The high bitrate savings of an NN model is typically directly related to the high complexity of the NN model. The model described in JVET-AF0041 has a complexity of 477 kMAC / pixel, i.e., 477,000 multiply- accumulate operations per pixel. There are also other measures of complexity, such as total model size in terms of stored parameters.

[0018] A Low Operation Point (LOP) architecture NN loop filter is currently used in a JVET EE test (EE1-1.0). In the LOP architecture, the input patch size is 144x144 and there is a final cropping step that crops 8 pixels from each side of the output luma patch and 4 pixels from each side of each chroma patch (which has half the size of the luma patch). Hence the final patch size after cropping in the output is equal to 128x128 for luma and 64x64 for chroma branch. The complexity of this current model is about 17 kMULT per sample. This means that about 17,000 multiplications are performed for calculating one sample value in the output of the neural network loop filter architecture.

[0019] A Very Low Operation Point (VLOP) NN loop filter is another NN loop filter tested in JVET. The architecture of VLOP is similar to LOP, but the complexity is only 5 kMAC / pixel. Compared to LOP, VLOP has a smaller number of backbone blocks and channels.

[0020] In JVET standardization, development and study of codecs is done using a common test conditions (CTC). The CTC specifies how a codec under test should be configured andwhat test sequences that should be used. Keeping the test conditions static enables apples-to- apples evaluations, but with the drawback that configurations outside the CTC are not tested much and the codec performance may become too optimized towards the CTC.

[0021] Intra Prediction Modes Based on Neural Networks

[0022] In JVET-J0037 contribution, intra prediction modes to generate an intra-picture prediction signal on a rectangular block were proposed. The proposed intra-prediction modes were based on fully connected neural networks (see FIG. 1) with several layers which were adding to the complexity compared to the prior art. To partially deal with the above mentioned complexity increase one remedy used in JVET-J0037 was that for a given block shape, all predictors shared all but the last layer in the neural network.

[0023] FIG. 1 illustrates a prediction of MxN intra block from reconstructed samples using a neural network from JVET-J0037. As shown in FIG. 1, the input to the neural network is the reconstructed samples on the top and left of an MxN patch and the output of the neural network is the predicted samples in the MxN patch. Basically, the solution provided in JVET- J0037 is for intra prediction purposes where it takes in sample values from outside a patch in order to predict sample values for the patch.

[0024] In another recent contribution JVET-AB0149, neural networks are used for mapping the intra prediction modes.

[0025] As shown in FIG. 2, the neighboring samples of the block Y are used as input to the network. The network gives three outputs, F, grpldx i, and repldx, where F undergoes postprocessing to generate Y_hat, grpldx_i (i=l or 2) denotes the index of LFNST kernel and whether transpose is applied during the primary transform, and repldx corresponds to the intra prediction mode (Planar, DC, or directional intra prediction). The preprocessing includes removing mean, scaling, and reshaping X into a vector. The postprocessing consists of reshaping the vector Y into a block of w x h, scaling and adding mean, and clipping the values to [0, 2b-l], Different networks are used for different block sizes. In JVET-AB0149, there are 7 networks with sizes {4 x 4, 8 x 4, 16 x 4, 32 x 4, 8 x 8, 16 x 8, 16 x 16}.SUMMARY

[0026] Certain challenges presently exist. For example, high complexity is a major issue with current NN based solutions when used in video codecs. This is due to the fact that a videodecoder typically must be able to output 60 frames per second. Also, video content is often something that is consumed for a long period of time. Hence a certain number of multiplications per pixel quickly becomes many multiplications per hour, draining battery life and also risking making the device overheat.

[0027] Therefore, it is desired to keep the complexity down for the neural network processing that is required by the decoder. Model complexity for neural networks in an image or video codec is usually quantified as the number of multiply-accumulate operations per pixel kMAC / pix (or kMAC for short) or the number of multiplications per pixel kMULT / px (or kMULT for short).

[0028] Another reason for keeping down complexity is to make it feasible to implement the decoder without too much cost. As an example, even if it were possible to run the decoder without draining the battery, this would still not be economically feasible if the amount of silicon surface needed to implement the decoder would be too expensive.

[0029] Another problem with existing technology is high energy consumption when training the neural network. This is directly related to the complexity of the network and is an important factor. Existing solutions are encouraged to improve their energy consumptions. In some embodiments, training is only done off-line and the encoder and decoder only do inference (“running” an already trained neural network). In other embodiments, training is done in the encoder on the sequence to be compressed. In such cases, energy consumption during training will have a direct impact on the energy consumption of the encoder.

[0030] Existing technology has a problem in that the complexity of the decoder in terms of kMAC / pix is too high to implement in some devices. Existing solutions are also encouraged to improve their complexity / performance tradeoff. This can be achieved for instance by reducing complexity for the same gain or increasing the performance for the same complexity. Performance for a codec is defined by increases in compression efficiency, often referred to simply as “gain” and is often measured as the Bjontegaard-delta rate (BDR) against an anchor. As an example, a BDR of -1% means that the same PSNR distortion can be reached with 1% fewer bits.

[0031] Current NN-based loop filters are trained so that they can handle all types of input patches using one set of trained parameters. For instance, for the case of NN based loop filters, one model with millions of parameters is trained to reduce deblocking artifacts andimprove the visual quality of all types of input patches, regardless of the content of the patch. In this approach all the input patches go through the same network layers and the processing will require the same number of operations. This is true for all HOP, LOP and VLOP operation points for the NN based loop filters currently under exploration in EEL This “one solution for all” approach could impose unnecessary complexity for improving the quality of some patches that for instance have less severe artifacts or have particular frequency components or texture properties. Also, it could be the case that the “one solution for all” approach for a NN-based filter for image or video with complexity of C has suboptimal performance (lower gain, higher kMAC) depending on the choice of the architecture, the number of parameters and the training process of the neural network model.

[0032] Accordingly, in one aspect, there is provided a method for image processing (e.g. image encoding and / or image decoding). The method includes obtaining a first input patch comprising a first set of one or more processed samples within a first region of an image (e.g., a block of samples from a picture of a video). The method also includes processing the first input patch using a first part of a neural network-based, NN-based, filter to produce a first set of NN outputs (e.g., a set of one or more channels or a set of one or more latent variables). The method also includes obtaining (e.g., producing or selecting) a first output patch based on a determined type of the first input patch, the first output patch comprising a second set of one or more processed samples within the first region of the image. Obtaining the first output patch based on the determined type of the first input patch comprises: 1) based on the determined type, selecting at least one NN from a set of two or more NNs (a.k.a., NN bank) and producing the first output patch using the selected NN(s) and the first set of NN outputs, where the set of NNs is part of a second part of the NN based filter, 2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs, where the set of NN layers is part of a second part of the NN based filter, 3) based on the determined type, selecting at least one set of weights (e.g., selecting at least one set of weights from a weight bank) and producing the first output patch using the selected set of weights, at least one NN, and the first set of NN outputs, or 4) using a set of two or more NNs and the first set of NN outputs to produce two or more candidate output patches and selecting, based on the determined type, at least one of the candidate output patches, wherein, if a single candidate output patch is selected, then the first output patch is the selected candidate output patch, otherwise the first output patch is derived using the selected candidate output patches.

[0033] In another aspect, there is provided an apparatus for performing any of the methods disclosed herein.

[0034] In another aspect, there is provided a computer program comprising instructions which when executed by processing circuitry cause the processing circuitry to perform the method of any one of the above embodiments.

[0035] In a different aspect, there is provided a carrier containing the computer program of the above embodiment, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0036] An advantage of some embodiments disclosed herein is that they improve the complexity / performance trade-off. This is achieved by one or more of the following: i) lowering the complexity as a result of all paths having similar or smaller complexity compared to the “one solution for all” approach, ii) improving the performance as a result of larger number of parameters in all the separate paths together compared to the number of parameters in “one solution for all” approach. Another advantage of the embodiments is providing the possibility of more specialized training of each separate branch of the neural network. This can be beneficial when the input space (the input patches) can be classified into clearly distinct groups that can benefit from specialized filtering processes.BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0038] FIG. 1 shows a prediction of MxN intra block from reconstructed samples using a neural network from JVET-J0037.

[0039] FIG. 2 shows neighboring samples of a block Y are used as input to a NN.

[0040] FIG. 3 illustrates a system according to some embodiments.

[0041] FIG. 4 is a schematic block diagram of an encoder according to an embodiment.

[0042] FIG. 5 is a schematic block diagram of a decoder according to an embodiment.

[0043] FIG. 6 illustrates an NN-based filter according to an embodiment.

[0044] FIG. 7 illustrates an NN-based filter according to an embodiment.

[0045] FIG. 8A illustrates an NN-based filter according to an embodiment.

[0046] FIG. 8B illustrates blending two patches.

[0047] FIG. 9 illustrates an NN-based filter according to an embodiment.

[0048] FIG. 10 illustrates an NN-based filter according to an embodiment.

[0049] FIG. 11 A illustrates an NN-based filter according to an embodiment.

[0050] FIG. 11B illustrates an NN-based filter according to an embodiment.

[0051] FIG. 12 illustrates prior art processing.

[0052] FIG. 13 illustrates an NN-based filter according to an embodiment.

[0053] FIG. 14 illustrates conv2D layer according to an embodiment.

[0054] FIG. 15 illustrates that an NN can be trained by selectively freezing weights and killing derivatives during training.

[0055] FIG. 16 illustrates a training method according to an embodiment.

[0056] FIG. 17 illustrates an example how control signals can depend continuously on a quantization parameter (QP).

[0057] FIG. 18 shows using min-pooling to find the best partitioning of the training data.

[0058] FIG. 19 is a flowchart illustrating a process according to some embodiments.

[0059] FIG. 20 is a block diagram of an apparatus according to some embodiments.DETAILED DESCRIPTION

[0060] FIG. 3 illustrates a system 300 according to an embodiment. System 300 includes an encoder 302 and a decoder 304, wherein, in the example shown, encoder 302 is in communication with decoder 304 via a network 110 (e.g., the Internet or other network). Encoder 302 encodes a source video sequence 301 (e.g., encodes blocks of units of pictures of video sequence 301) into a bitstream comprising an encoded video sequence (e.g., encoded blocks) and transmits the bitstream to decoder 304 via network 308. In some embodiments, encoder 302 is not in communication with decoder 304, and, in such an embodiment, rather than transmitting bitstream to decoder 304, the bitstream is stored in a data storage unit 390 and decoder 303 can retrieve the bitstream from data storage unit 390. Decoder 304 decodes the pictures included in the encoded video sequence to produce video data for display and / or for further image processing (e.g., a machine vision task). Accordingly, decoder 304 may be part of a device 303 having an image processor 305 and / or a display 306. The imageprocessor 305 may perform machine vision tasks on the decoded pictures. The device 303 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0061] FIG. 4 illustrates functional components of encoder 302 according to some embodiments. It should be noted that encoders may be implemented differently so implementation other than this specific example can be used. Encoder 302 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block and a prediction block (i.e., the output of a selector 251, which is either an inter-prediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intra-prediction block output by an intra predictor 249). Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. A quantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transform coefficients which are then encoded into the bitstream by encoder 244 (e.g., an entropy encoder) and the bitstream with the encoded transform coefficients is output from encoder 302. Next, encoder 302 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstruction picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, where it can then be used by the inter predictor 250 to produce an inter-prediction block for the next picture to be processed. LF stage 267 may include three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAG) filter, and iii) an Adaptive Loop Filter (ALF).

[0062] FIG. 5 illustrates functional components of decoder 304 according to some embodiments. It should be noted that decoder 304 may be implemented differently so implementations other than this specific example can be used. Decoder 304 includes a decoder module 361 (e.g., an entropy decoder) that decodes from the bitstream quantized transform coefficient values of a block. Decoder 304 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block.This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter-prediction block or an intra-prediction block. The reconstructed block is stored in a RPB 365. The inter-prediction block is generated by the inter-prediction module 350 and the intra-prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies loop filtering and the final decoded picture may be stored in a decoded picture buffer (DPB) 368 and output to image processor 105. Pictures are stored in the DPB for two primary reasons: 1) to wait for picture output and 2) to be used for reference when decoding future pictures.

[0063] As noted above, conventional NN-based components, such as conventional NN-based in-loop filters, have several disadvantage, including high complexity.

[0064] Accordingly, this disclosure introduces, in one embodiment, a hierarchy in the computationally intensive process of NN-based filtering of video patches so that not all patches see exactly the same path through the NN-based filter. In one example, all patches go through certain layers (e.g., layers Li to Li), while patches assigned to group A pass through layers L(i+i) till Ln but patches assigned to group B pass through layers L’(i+i) till L’k. That is, in an embodiment, at least two different patches go through partially different layers in the NN-based filter. The embodiments provide a better complexity / performance trade-off compared to the scenario where all the patches must go through the worst-case complexity path.

[0065] In one implementation, instead of one computationally costly NN process that fits all the input patches, the process consists of a common part applied to all patches and at least one non-common part applied to only a subset of the patches. The rationale is that the total computational complexity for processing all the patches could then be smaller.

[0066] In one scenario, the “one solution for all” approach with complexity of C has suboptimal performance (lower gain, higher kMAC) compared to an embodiment with two partially different paths with complexity Cl and C2, where Cl < C and C2 < C. One clear case in this scenario is when the number of parameters in the network with complexity of C is less than the number of parameters in networks with complexity of Cl and C2 combined. In this case the learning capacity for networks with complexity of Cl and C2 combined is potentially larger than the network with complexity of C, and so the performance of thenetwork with complexity of Cl and C2 combined could potentially be higher than the learning capacity of network with complexity of C. As a result, the embodiment with Cl and C2 could provide a better gain and a lower complexity in comparison with the “one solution for all” approach with complexity of C.

[0067] In one particular embodiment, a NN loop filter is provided where the NN loop filter consists of two parts: a first part that is the common process applied to all input patches, and a second part which is different for at least two different input patches.

[0068] In some embodiments, the second part of the loop filter is a filter bank consisting of two different NNs. In this embodiment a control input determines which of the NNs in the filter bank are going to be used in the second part of the loop filter for each input patch to the loop filter.

[0069] In some embodiments, at least two processes e.g. Pl and P2 in second part 602 are sequential. In this embodiment each input patch goes through the fist part of the filter which is the common process and in second part 602, a control input determines if the patch is going through the Pl process or Pl and then P2 process sequentially.

[0070] In another embodiment, there is a further processing after second part 602 e.g. for blending results from more than one filter in the filter bank in order to construct the final output.

[0071] In one embodiment, the information about which of the processes in second part 602 is going to be applied to an input patch to the loop filter is signalled in the bitstream. In yet another embodiment, this information is determined in the common process for instance extracted from the sample values in the input patch to the loop filter.

[0072] In another aspect there is provided a method for training an NN-based loop filter that uses, for example, min-pooling, max-pooling or freezing weights and biases for unchosen branches and setting the backwards derivative to zero for the frozen branches.

[0073] Additional Details Regarding Improved NN-based Filter Systems

[0074] FIG. 6 illustrates an NN-based filter 600 (or “filter” for short) according to an embodiment. Filter 600 includes a first part 601 and a second part 602. Filter 600 is configured such that first part 601 processes all input patches 611. In one embodiment, second part 602, for some input patches but not others (e.g., patches of type “B”, but notpatches of type “A”) filters the output 613 of first part 601. In another embodiment, second part 602 is configured to i) process output 613 using a first process when input patch 611 is a patch of type A but ii) process output 613 using a different process when input patch 611 is a patch of type B. The output 613 of first part 601 may comprise a set of patches (e.g., a set of two or more channels or latent variables). Second part 602 may also receive other inputs, such as the input patch 611 and anew input 612 (denoted “input2”) and process output 613 of first part 601 using one or more of these other inputs.

[0075] In one embodiment, second part 602 decides whether or not to process (or how to process) output 613 of first part 601 based on a type value associated with the input patch 611. This type value may be included in input2 612 or may be provided to second part 602 by first part 601, which may derive the type value or obtain it from the bitstream. That is, the type value may be used as a control signal to choose a particular type-based process in the second part 602, or be used as an input that affects the process in second part 602 and makes the process particular to the type value. The type value may describe the type of the input patch 611 or may identify a particular processing unit (e.g., NN filter) of second part 602. The type value may be provided to the second part of the process as a separate input (e.g. input2) or included in the output 613 of the first part 601. For example, in one embodiment, the type value is extracted in the first part 601, provided to second part 602, and then, as described with reference to FIGs. 7-10, is used to determine which of the filters of a filter bank are used for the second step processing.

[0076] In one example, input patch 611 is a 2D array of pixels (e.g., an NxM matrix of sample values) and filter 600 is an NN-based loop filter where first part 601 consists of one or more convolutional layers which processes input patch 611 to produce one or more output channels, where a channel is a 2D array of values. In this example, second part 602 is a typebased process depending on the type of the input patch being processed in the filter. This means that patchl and patch2 with two different patch types will be treated exactly the same in the first part 601 (i.e., the “common” part) and not the same in the second part 602. The patch type for each input patch may be given as input2 to the type-based process (second part of the filter), or in another example the type could be extracted from inputl as part of the common process and passed to second part 602 to be used in the type-based process in second part 602.

[0077] In another example, first part 601 performs an affine transform, e.g., a DCT transform on input patch 611. First part 601, in this example, includes a thresholding mechanism on different frequency elements of the DCT transform and the thresholding mechanism determines the type value for the input patch 611. For example, the type value for the input patch 611 is set to type A if the ratio of the lowest frequency component in the patch to the second lowest frequency component in the patch is smaller than a threshold, otherwise the type value is set to type B. This type value is then passed to second part 602, which in this example is a neural network consisting of one or more convolutional layers which takes in the type value as one of the inputs, and, in one embodiment, if the type value is type A processes the output 613 of first part 601 using one set of convolutional layers (or does no processing), otherwise processes the output 613 using a different set of convolutional layers. The output of second part 602 is an output patch 615 (or “output 615” for short) with sample values collocated with input patch 611. That is, input patch 611 comprises a first set of samples for a particular region of a picture and output patch 615 comprises of a second set of samples for the same particular region of the picture. For instance, input patch 611 may consist of a set of samples for a first rectangular region of the picture (e.g., an NlxMl region) wherein the particular region (e.g., an N2xM2 regions) is within the first rectangular region (i.e., N2 < N1 and M2 < Ml) and output patch 615 consists of a set of samples for the same N2xM2 region of the picture

[0078] In another example, first part 601 is a classifier and provides the class (i.e., type) of the input patch to second part 602. In another example the first part 601 is a classifier neural network that gives the probability of the patch being in each class.

[0079] Parallel Branches in Second Part 602

[0080] FIG. 7 illustrates an embodiment in which second part 602 comprises an NN-filter bank 702 - - i.e., a set of two or more NN filters (N number of filters is illustrated in FIG. 7). In one embodiment, input2 612 (e.g., the type value) is provided to second part 602 (or derived by second part 602) and a selector 704 of second part 602 uses input2 (e.g. ,the type value) to select one or more of the NN filters in the filter bank and then the selected NN filters process output 613 to produce output patch 615. The filter bank may be in the form of parallel NN-filters as shown in FIG. 7. If selector 704 selects just one of the filters of filter bank 702 to process output 613, then the final output 615 may be the output from the selected NN-filter. If, however, selector 704 selects two or more of the filters of filter bank 702 toprocess output 613, then the final output 615 may be based on one or more outputs from one or more of the selected NN filters, as, for example, described below.

[0081] Blending Filter Outputs to Produce Output 615

[0082] FIG. 8 A illustrates that second part 602, in addition to comprising filter bank 702 and selector 704, also includes a combiner 822. In one embodiment, combiner 822 functions to blend results from more than one branch (NN filter) of filter bank 702 to produce final output 615. That is, in one embodiment, output 615 is a blend of two or more NN filter outputs 801- 1, 801-2, ..., 801-N. This is useful, for example, when each filter in the filter bank is specialized in processing some aspects of the input and a final merging of the results is needed to provide the output 615. In this embodiment, like the embodiment shown in FIG. 7, not all the input patches necessarily go through the same filtering process except for the common processing step (i.e., first part 601.

[0083] As noted above, combiner 822 may perform a blending process to blend the NN filter outputs if more than one of the NN filters is selected to process output 613. In one example, the blending process is a simple weighting between two or more outputs of the filters in the filter bank. In another example the blending process is performed using a fully connected layer which is trainable or has fixed weights.

[0084] In one embodiment, two or more of the output 801-1 to 801-N are blended using a weight function. In one example the weight function is signalled and it specifies how much weight to give to each output 801-1 to 801-N when performing the blending.

[0085] FIG. 8B illustrates an example where output patch 615 is a blend of output 801-1 and output 801-2. In the example shown, the top left comer of the output patch 615 contains 0% of the output 801-1 and 100% of 801-2, the top right comer and the bottom left comers have 50% from output 801-1 and 50% from output 801-2, and the bottom right comer of the output patch contains 100% of output 801-1 and 0% of output 801-2. Other sample positions in this example get a blending weight combination of 801-1 and 801-2 linearly proportional to their distances from the comers of the patch. The weight function may be signalled in the bitstream or may be determined on the decoder side for instance as part of the common process.

[0086] Selecting Filter Outputs

[0087] FIG. 9 illustrates an embodiment in which each NN filter of filter bank 702 processes output 613 (e.g., one or more output channels) to produce an NN filter output and combiner 822 selects one or more NN filter outputs (e.g., filter output 801-1, filter output 801-2, ..., and / or filter output 801-N) and produces output 615 using the selected NN filter outputs as, for example, described above with reference to FIGs. 8A and 8B. That is, in this embodiment, output 615 is based on the selected NN filter outputs.

[0088] Filter Bank 702

[0089] In one embodiment, each filter of filter bank 702 is a convolutional NN. In one example, each convolutional NN of filter bank 702 is trained to remove deblocking artifacts in the input patch for patches with particular value or range of type values. In one example, the type is the QP associated with input patch 611. In another example the type is the ratio of the lowest to the second lowest frequency component in the patch, so that if the ratio is smaller than a threshold the type value is set equal to type A and if the ratio is equal or larger than a threshold the type value is set equal to type B.

[0090] Training of such a NN filter bank with several parallel NN filters could be done using min-pooling and making sure that the branches (i. e. , all the NN filters that comprise filter bank 702) all get a good amount of training data. One way to ensure a balance between the amount of data each branch is getting is by keeping the data allocation score for each branch and keep the score in balance by favouring the lower score branches.

[0091] In one example, the allocation of the training data to a particular branch is through mechanisms such as min-pooling. In another example, the training is performed for known classes, based on e.g. QP or based on frequency analyses of the input patch to the filter, or the ground truth data available for instance to the encoder.

[0092] The type value may be signalled in the bitstream or may be determined on the decoder side for instance in the first part 601.

[0093] Sequential Layers in Second Part 602

[0094] FIG. 10 illustrates an embodiment in which filter bank 702 of second part 602 includes a sequence of one or more NN filters 1021-1, 1021-2, ..., 1021-N. Second part 602 in this embodiment includes a selector 1002 that selects, based on for example input2, which one, if any, of the N NN filters will be employed to produce the output patch 615. Forexample, a type value (e.g., included in input2 or provided to second part 602 by first part 601 determines how selector 1002 will control switches 1011, 1012, 1013, 1014, and 1015, which will determine which, if any, of the NN filters will be employed to produce output 615. For instance, in one embodiment, selector 1002 can control switch 1011 such that filter 1 and / or a combiner 822 receives output 613, and, likewise can control switch 1012 such that: 1) the output of filter 1 goes only to combiner 822, 2) the output of filter 1 goes only to a filter 2, or 3) the output of filter 1 goes to both combiner 822 and filter 2. In this way, based on a type value, zero, one or more or all of filters 1021-1 to 1021-N can be employed to produce output 615.

[0095] In one embodiment each filter 1021-1 to 1021-N comprises one or more convolutional layers and the type value determines how many of these convolutional layers is to be applied to prepare the output 615.

[0096] As illustrated in FIG. 10, multiple filters may be employed to produce output 615 and in such a scenario combiner 822 receives the output from each filter. Hence, in one embodiment, output 615 is based on more than one of the filter outputs as, for example, described above with respect to FIGs. 8A and 8B.

[0097] Further processing modules may be included in second part, such as, for example, a processing module that process the output of combiner 822 and / or processes the output from one or more selected filters of the filter bank, such that the output of this further processing module is the final output 615.

[0098] This feature is illustrated in FIGs. 11 A and 1 IB, which illustrates second part 602 including an additional processing module 1102. Additionally, the above described embodiments may be combined with each other. The combination of embodiments is not limited to the provided examples.

[0099] Signalled patch type

[0100] In one embodiment, the type value is not determined by first part 601, but rather, for example, is signalled in a bitstream containing coded samples that are processed to produce input patch 611. In one example, NN-based filter 600 is a loop filter in an image or video codec, the type value for each patch of the image or video is signalled in the encoded bitstream, and input patch 611 may be a patch of a reconstructed picture before in-loop filtering or a patch of a filtered reconstructed picture before the final in-loop filter. In thisexample, the decoder, which includes NN-based filter 600, decodes from the bitstream a type value for each input patch to the loop filter, and the decoded type value is then used as an input parameter to the second part 602 of the fileting process. As described above, in one embodiment the type value parameter is used to control the filter selection process in the filter bank. In another variant of this embodiment, the type value is instead or in addition used as an input parameter to second part 602, e.g. as (part of) input2 to the filter bank.

[0101] Training

[0102] FIG. 12 shows a very small prior art NN-based filter 1200 that includes i) a first part 1201, which in this example includes a single 2D convolutional layer (conv2D), and ii) a second part 1202, which in this example also includes a single conv2D. Input patch 611 is fed into a conv2D 1201, which produces N channels of output 1213. Then the second conv2D 1202 uses these N channels 1213 as input to produce a corrected version of the input patch as output 1215. During training, an error between the desired output (for instance the original patch corresponding to input patch 611 and the output 1215 is backpropagated to update the weights and biases in the two conv2D layers 1201 and 1202.

[0103] FIG. 13 shows NN-based filter 600 according to a simplified embodiment to illustrate a training process according to an embodiment. In this embodiment, first part 601 includes a single conv2D layer that receives and process input patch 611 to produce N channels of output 613. Output 613 is fed into second part 602, which in this example is type-based and also consists of a single conv2D layer. Instead of producing one output, conv2D layer 602 produces two outputs (i.e., two channels in this embodiment); a first channel cO 1308 and a second channel cl 1309. Both of these channels cO and cl are then forwarded to a selection mechanism 1310 that select as output 615 either cO or cl. A question is now how this network should be trained using the desired output. The selection mechanism 1310, which in this case acts as a multiplexor by selecting one of cO or cl, is not a common neural network layer that packages such as PyTorch can handle out-of-the-box. There are, however, several ways in which this network may be trained using regular packages such as PyTorch.

[0104] It should be understood that the example in FIG. 13 has been made small for the purposes of clarity of explaining the training. In a real implementation, it is likely that first part 601 will consist of many conv2D layers, and that likewise second part 602 will also consist of many layers.

[0105] FIG. 14 shows a more detailed description of the second conv2D layer 602 from FIG. 13. For simplicity, in this figure it is assumed that the N=2 for the N channels of output 613 from conv2D 601 in FIG. 13. Channel 1403a represents channel 0 of 613 and 1403b represents channel 1 of 613.

[0106] Also for simplicity, it is further assumed that the conv2D 602 from FIG. 13 has the kernel size 3x3. A kernel is a matrix or matrix like structure containing weights. The convolution outputs two channels, cO 1308 and cl 1309. To calculate the sample marked with gray in cO 1308, the gray area in 1403a is element- wise multiplied with the weights waO 1414, and this result is added to the result of element-wise multiplying the gray area in 1403b with the kernel wbO 1413. A bias term (not shown) is then added, and the network may be fed through an activation function (not shown) to produce the sample marked with gray in cO 1308. This process is then repeated for all the samples in cO 1308 to produce the first channel. Analogously, cl 1309 is produced by convolving 1403a with weights wal 1412, 1403b with weights wbl 1411 and summing them together with a bias term and feeding that through an activation function to get cl 1309.

[0107] As should be clear from this picture, the calculation of cO 1308 is independent from the calculation of cl 1309 in the sense that cO is not needed as an input to calculate cl or vice versa. Hence in a decoder, if the type is known to be the one associated with cO, it only needs to calculate cO and can avoid calculating cl, which saves computation. Similarly, if the type is known to be the one associated with cl, it only needs to calculate cl and can avoid calculating cO.

[0108] FIG. 15 illustrates that the network 600 can be trained by selectively freezing weights and killing derivatives during training. More specifically, FIG. 15 shows a training example with type 0 training data, i.e., training data associated with the output cO 1508. In this case, the loss is calculated for both cO and cl, but the weights wbl 1511 and wal 1512 associated with output cl 1509 are frozen, as is the bias term (not shown). Also, the derivatives moving backwards from weights 1511 and 1512 are killed (set to zero). This means that the earlier layers, such as the conv2D layer 601 in FIG. 13, are not affected by the difference between the output cl 1509 and the desired output. Likewise, for type 1 training data, weights wbO 1513 and waO 1514 are frozen and its backward propagating derivative is set to zero.

[0109] Another way of training the neural network is to implement the selection mechanism 1310 from FIG. 13. One way of doing this is to multiply the output channel corresponding to the selected type by 1 and the other one by 0.

[0110] An example of this is shown in FIG. 16. Here, two outputs cO 1608 and cl 1609 are created. That is, output cO 1608 is created using weights wbO 1613 and waO 1614 and output cl 1609 is created using weights wbl 1611 and wal 1612. Assume that the current training sample is of type 0. This means that the corresponding output cO 1608 is multiplied by the signal mO 1630 which is equal in this case to 1 to produce the modified output cO’ 1620, which in this case is identical to cO. However, the other output cl 1609 is multiplied by signal ml, which in this case is equal to 0, to produce a modified output cl’ 1621 which is completely zero. The two modified outputs cO’ 1620 and cl’ 1621 are then added together to form the final output 1606. The multiplication by 0 will mean that no error is going to be backpropagated from channel cl.[OHl] The control signals mO and ml must also be calculated. One possibility is to calculate them as shown in the table below.

[0112] A generalization to three outputs and three types could be as shown in the table below.

[0113] This type of coding is usually referred to as one-hot-encoding, since the signal corresponding to the correct type is ‘hot’ (i. e. , 1) whereas all other signals are 0.

[0114] In some circumstances, however, it may be desirable to be able to specify not only the cases when something is 100% type 0 or 100% type 1, but something in between. As an example, assume one has training data for material compressed with quantization parameter (QP) equal to 22, 27, 32 and 37. It can be said that typeO is when QP=22, typel is when QP=27, type2 is when QP=32 and type3 is when QP=37. Now say the training data is increased to also include data that has QP 24. This is somewhere in between QP=22 and QP=27. Since 24 = 0.6 * 22 + 0.4 * 27, one could say that QP=22 should be 60% typeO and 40% typel.

[0115] FIG. 17 shows an example of such a set of control signals m0(QP), ml(QP), m2(QP) and m3(QP).

[0116] For the simple cases, one can verify that these control signals work as expected. As an example, QP=22 is associated with type 0, so should have m0=l and all the others ml=m2=m3=0. Likewise, QP=27 gives ml=l and m0=m2=m3=0, etc. However, in the case when QP = 24, one can see in the top left diagram 2210 that mO will be positive (it will be equal to 0.6) and in the bottom left diagram 2211 that ml will also be positive (it will be equal to 0.4). One can also see that the other outputs m2 and m3 will be equal to zero. This means that the network will create a blend of the outputs for typeO (QP=22) and typel (QP=27) and the loss during training will be calculated from the difference between this blend and the desired signal.

[0117] In detail, the functions m0(QP), ml(QP), m2(QP) and m3(QP) can be written as: m0(QP) = (l / 5)*min( ReLU(QP-17), ReLU(27-QP) ) ml(QP) = (l / 5)*min( ReLU(QP-22), ReLU(32-QP) )m2(QP) = (l / 5)*min( ReLU(QP-27), ReLU(37-QP) ) m3(QP) = (l / 5)*min( ReLU(QP-32), ReLU(42-QP) ) where the function ReLU(x) is defined as x if x>0 and 0 otherwise.

[0118] These control signals can be used not only during training, but also during inference. As an example, if a patch has QP=24, it is possible to calculate the control signals according to the above resulting in m0=0.6, ml=0.4, m2=m3=0, and then use those to produce a blended output during inference.

[0119] In some cases, it is not known during training which is the best way to partition the outputs. As an example, for the network depicted in FIG. 14, it may be the case that the designer thinks the best way is to divide up the input data via QP: As an example; patches with low QP are designated to be type 0 and will train the output cO, whereas patches with high QP are designated to be type 1 and will train the output cl. However, perhaps it is better to have bright patches be of type 0 and dark patches to be of type 1. Instead of having the designer of the network make educated guesses of what way to partition the training data is the best, this can instead be found out implicitly in the training process, for instance in the way illustrated in FIG. 18.

[0120] In FIG. 18, training data 2332 is always fed to both outputs cO 2308 and cl 2309 and two errors (losses) are calculated: error 02330 is the loss between the output cO 2308 and the training data 2332, and error 1 2331 is the loss between the output cl 2309 and the training data 2332. These two losses are fed into a min-function, and the final loss is calculated as: final_loss = min ( errorO, errorl ).

[0121] The network can now be trained by minimizing final loss. This is a form of min- pooling; only the error of the best output is going to be affecting the final loss, and hence the error is going to be backpropagated only through the parts of the network that were responsible for producing that error. As an example, if error 0 is smaller than error 1, that means that cO was the better output. Since min-pooling happens in 2333, the weights only involved in producing cO, namely weights wbO 2313 and weights waO 2314, will be affected during backpropagation. Also, earlier weights to produce the signals 2303a and 2303b will be affected. By contrast, the weights only involved with producing cl, namely weights wbl 2311 and weights wal 2312, will not be affected. This way, the weights responsible for cOwill only be trained on patches where output cO is better than the other outputs (in this case there is only one other output, cl), and will hence “specialize” in those types of patches.

[0122] One difficulty with this way of training is that it can happen by pure chance that cO gets chosen in the beginning of the training more often than cl. This may lead to an unbalanced training, where all or almost all of the training data goes to the weights responsible to produce cO. This means that the weights responsible to produce cl gets starved for training data, with poor performance of the cl output as a result, further lowering the chance that cl will win in the min-pooling. In order to avoid this, it may be necessary to multiply the error of the starved output by a factor smaller than one so that it again starts to win the min-pooling. An alternative is to multiply the error selected most often by a number larger than one. In the case where more than two outputs are generated, it may be necessary to multiply every error with a scalar number before calculating min(). An example could be to multiply it by pk, where pk is the probability that this output will win the min-pooling. Randomness can also be inserted into this process so that it is multiplied by md*pk rather than by pk where md is a random number between 1 and 0.

[0123] When using a network trained this way during inference, it may not be obvious for the encoder which output will produce the best result. Therefore, one way to do the encoding is to try all possible outputs (cO, cl, c2, etc.) and then signal to the decoder which one was the best. This may be costly in the encoder. Therefore, an alternative is to train another neural network to predict which output is the best, for instance by assigning a probability to each output. Then the encoder can choose to encode (and signal) only the one with the highest probability. Alternatively, the decoder can run the same network on the receiving side so that it knows which output had the highest probability, and then the chosen output needs not to be signalled. Alternatively, the encoder can try encoding the four outputs corresponding to the highest probabilities, see which one is actually best (it may not be the one with the highest probability), use the best one and signal that to the decoder. If the decoder also runs the same network, only which of the four highest ones needs to be signalled.

[0124] FIG. 19 is a flowchart illustrating process 1900, according to some embodiments, for image processing (e.g., image encoding or image decoding). Process 1900 may begin in step S1902. Step s!902 comprises obtaining a first input patch (e.g., input patch 611) comprising a first set of one or more processed samples within a first region of an image (e.g., a block of samples from a picture of a video). Step sl904 comprises processing the firstinput patch using a first part (e.g., part 601) of a neural network-based, NN-based, filter (e.g., filter 600) to produce a first set of NN outputs (e.g., outputs 613, such as, for example, a set of one or more channels or a set of one or more latent variables). Step si 906 comprises obtaining (e.g., producing or selecting) a first output patch (e.g., output patch 615) based on a determined type of the first input patch, the first output patch comprising a second set of one or more processed samples within the first region of the image. Obtaining the first output patch based on the determined type of the first input patch comprises:

[0125] 1) based on the determined type, selecting at least one NN from a set of two or more NNs (a.k.a., NN bank) and producing the first output patch using the selected NN(s) and the first set of NN outputs, where the set of NNs is part of a second part (e.g., part 602) of the NN based filter,

[0126] 2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs, where the set of NN layers is part of a second part of the NN based filter,

[0127] 3) based on the determined type, selecting at least one set of weights (e.g., selecting at least one set of weights from a weight bank (i.e., two or more sets of weights)) and producing the first output patch using the selected set of weights, at least one NN, and the first set of NN outputs, or

[0128] 4) using a set of two or more NNs and the first set of NN outputs to produce two or more candidate output patches and selecting, based on the determined type, at least one of the candidate output patches, wherein, if a single candidate output patch is selected, then the first output patch is the selected candidate output patch, otherwise the first output patch is derived using the selected candidate output patches.

[0129] In some embodiments, the NN-based filter is an NN-based loop filter.

[0130] In some embodiments, the first part of the NN-based filter comprises a plurality of convolutional layers and the first set of NN outputs comprises a plurality of channels.

[0131] In some embodiments, the second part of the NN-based filter obtains a type value indicating the type of the first input patch.

[0132] In some embodiments, the first part of the NN-based filter provides the obtained type value to the second part of the NN-based filter.

[0133] In some embodiments, the obtained type value is inferred from one or more encoded parameters from a bitstream (e.g., a bitstream comprising an encoded version of the image).

[0134] In some embodiments, obtaining the first output patch comprises selecting at least one NN from the set of two or more NNs, each NN included in the set of two or more NNs is associated with one or more type values, and selecting at least one NN from the set of two or more NNs comprises selecting a first NN from the set of two or more NNs as a result of determining that the first NN is associated with a type value that matches the obtained type value.

[0135] In some embodiments, each NN included in the set of two or more NNs is a convolutional NN.

[0136] In some embodiments, each convolutional NN is trained to remove artifacts.

[0137] In some embodiments, the obtained type value is a quantization parameter, QP, value associated with the first input patch.

[0138] In some embodiments, the obtained type value is, or is based on, a ratio of a lowest frequency component in the first input patch to a second lowest frequency component in the first input patch.

[0139] In some embodiments, the obtained type value is type A if the ratio is less than a threshold or the obtained type value is type B if the ratio is greater than the threshold.

[0140] In some embodiments, the first part of the NN-based filter performs an affine transform on the first input patch, and the first part selects type A as the obtained type value if the ratio is less than a threshold or less than or equal to the threshold, otherwise the first part selects type B as the obtained type value.

[0141] In some embodiments, obtaining the first output patch comprises selecting at least a first NN and a second NN from the set of two or more NNs, the first NN uses the first set of NN outputs to produce a first intermediate patch, the second NN uses the first set of NN outputs to produce a second intermediate patch, and obtaining the output patch comprises blending at least the first and second intermediate patches.

[0142] In some embodiments, process 1900 also includes, prior to obtaining the first output patch based on the determined type of the first input patch, determining the type of the first input patch. In some embodiments, determining the type of the first input comprises obtaining a type value indicating the type of the first input patch. In other embodiments, determining the type of the first input comprises obtaining a type value indicating the type of the first input patch.

[0143] FIG. 20 is a block diagram of an apparatus 2000 for implementing encoder 802 and / or decoder 804, according to some embodiments. When apparatus 2000 implements encoder 802, apparatus 2000 may be referred to as an encoder apparatus, and when apparatus 2000 implements decoder 804, apparatus 2000 may be referred to as a decoder apparatus. As shown in FIG. 20, apparatus 2000 may comprise: processing circuitry (PC) 2002, which may include one or more processors (P) 2055 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (e.g., apparatus 2000 may be a distributed, cloud computing system comprising two or more computers or a monolithic computing system consisting of a single computer); at least one network interface 2048 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 2045 and a receiver (Rx) 2047 for enabling apparatus 2000 to transmit data to and receive data from other nodes connected to a network 100 (e.g., an Internet Protocol (IP) network) to which network interface 2048 is connected (physically or wirelessly) (e.g., network interface 2048 may be coupled to an antenna arrangement comprising one or more antennas for enabling apparatus 2000 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 2008, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 2002 includes a programmable processor, a computer readable storage medium (CRSM) 2042 may be provided. CRSM 2042 may store a computer program (CP) 2043 comprising computer readable instructions (CRI) 2044. CRSM 2042 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 2044 of computer program 2043 is configured such that when executed by PC 2002, the CRI causes apparatus 2000 to perform steps described herein (e.g.,steps described herein with reference to the flow charts). In other embodiments, apparatus 2000 may be configured to perform steps described herein without the need for code. That is, for example, PC 2002 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0144] Summary of Various Embodiments

[0145] Al. A method (e.g., method 1900) for image processing, the method comprising: obtaining a first input patch (e.g., input patch 611) comprising a first set of one or more processed samples within a first region of an image (e.g., a block of samples from a picture of a video); processing the first input patch using a first part 601 of a neural network-based, NN-based, filter to produce a first set of NN outputs 613 (e.g., a set of one or more channels or a set of one or more latent variables); obtaining (e.g., producing or selecting) a first output patch (e.g., output patch 615) based on a determined type of the first input patch, the first output patch comprising a second set of one or more processed samples within the first region of the image, wherein obtaining the first output patch based on the determined type of the first input patch comprises: 1) based on the determined type, selecting at least one NN from a set of two or more NNs (a.k.a., NN bank) and producing the first output patch using the selected NN(s) and the first set of NN outputs 613, where the set of NNs is part of a second part 602 of the NN based filter, 2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs 613, where the set of NN layers is part of a second part 602 of the NN based filter, 3) based on the determined type, selecting at least one set of weights (e.g., selecting at least one set of weights from a weight bank) and producing the first output patch using the selected set of weights, at least one NN, and the first set of NN outputs 613, or 4) using a set of two or more NNs and the first set of NN outputs 613 to produce two or more candidate output patches and selecting, based on the determined type, at least one of the candidate output patches, wherein, if a single candidate output patch is selected, then the first output patch is the selected candidate output patch, otherwise the first output patch is derived using the selected candidate output patches.

[0146] A2. The method of embodiment Al, wherein the NN-based filter is an NN-based loop filter.

[0147] A3. The method of embodiment Al or A2, wherein the first part of the NN-based filter comprises a plurality of convolutional layers and the first set of NN outputs comprises a plurality of channels.

[0148] A4. The method of any one of embodiments Al -A3, wherein obtaining the first output patch based on the determined type of the first input patch comprises: 1) based on the determined type, selecting at least one NN from a set of two or more NNs and producing the first output patch using theselected NN(s) and the first set of NN outputs 613, where the set of two or more NNs is part of a second part 602 of the NN based filter, or 2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs 613, where the set of two or more NN layers is part of a second part 602 of the NN based filter, and the second part of the NN-based filter obtains a type value indicating the type of the first input patch.

[0149] A5. The method of embodiment A4, wherein the first part of the NN-based filter provides the obtained type value to the second part of the NN-based filter.

[0150] A6. The method of embodiment A4, wherein the obtained type value is inferred from one or more encoded parameters from a bitstream (e.g., a bitstream comprising an encoded version of the image).

[0151] A7. The method of any one of embodiments A4-A6, wherein obtaining the first output patch comprises selecting at least one NN from the set of two or more NNs, each NN included in the set of two or more NNs is associated with one or more type values, and selecting at least one NN from the set of two or more NNs comprises selecting a first NN from the set of two or more NNs as a result of determining that the first NN is associated with a type value that matches the obtained type value.

[0152] A8. The method of embodiment A7, wherein each NN included in the set of two or more NNs is a convolutional NN.

[0153] A9. The method of embodiment A8, wherein each convolutional NN is trained to remove artifacts.

[0154] A10. The method of any one of embodiments A7-A9, wherein the obtained type value is a quantization parameter, QP, value associated with the first input patch.

[0155] All. The method of any one of embodiments A7-A9, wherein the obtained type value is, or is based on, a ratio of a lowest frequency component in the first input patch to a second lowest frequency component in the first input patch.

[0156] A12. The method of embodiment Al l, wherein the obtained type value is type A if the ratio is less than a threshold, or the obtained type value is type B if the ratio is greater than the threshold.

[0157] A13. The method of embodiment A12, wherein the first part of the NN-based filter performs an affine transform on the first input patch, and the first part selects type A as the obtained type value if the ratio is less than a threshold or less than or equal to the threshold, otherwise the first part selects type B as the obtained type value.

[0158] A14. The method of any one of embodiments A7-A13, wherein obtaining the first outputpatch comprises selecting at least a first NN and a second NN from the set of two or more NNs, the first NN uses the first set of NN outputs to produce a first intermediate patch, the second NN uses the first set of NN outputs to produce a second intermediate patch, and obtaining the output patch comprises blending at least the first and second intermediate patches (e.g., obtaining the output patch comprises blending the first and second intermediate patches).

[0159] A15. The method of any one of claims A1-A14, wherein the method also includes, prior to obtaining the first output patch based on the determined type of the first input patch, determining the type of the first input patch. In one embodiment, determining the type of the first input comprises obtaining a type value indicating the type of the first input patch. In another embodiment, determining the type of the first input comprises obtaining a type value indicating the type of the first input patch.

[0160] Bl. A computer program 2043 comprising instructions 2044 which when executed by processing circuitry 2002 cause the processing circuitry 2002 to perform the method of any one of the above embodiments.

[0161] B2. A carrier containing the computer program of embodiment Bl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium 2042.

[0162] Cl. An apparatus 2000 for image processing, the apparatus being configured to perform a method comprising: obtaining a first input patch comprising a first set of one or more processed samples within a first region of an image; processing the first input patch using a first part 601 of a neural network-based, NN-based, filter to produce a first set of NN outputs 613 (e.g., a set of one or more channels or a set of one or more latent variables); obtaining (e.g., producing or selecting) a first output patch (e.g. output patch 615 based on a determined type of the first input patch, the first output patch comprising a second set of one or more processed samples within the first region of the image, wherein obtaining the first output patch based on the determined type of the first input patch comprises: 1) based on the determined type, selecting at least one NN from a set of two or more NNs (a.k.a., NN bank) and producing the first output patch using the selected NN(s) and the first set of NN outputs 613, where the set of NNs is part of a second part 602 of the NN based filter, 2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs 613, where the set of NN layers is part of a second part 602 of the NN based filter, 3) based on the determined type, selecting at least one set of weights (e.g., selecting at least one set of weights from a weight bank) and producing the first output patch using the selected set of weights, at least one NN, and the first set of NN outputs 613, or 4) using a set of two or more NNs and the first set of NN outputs 613 to produce two or more candidate output patches and selecting, based on the determined type, at least one of thecandidate output patches, wherein, if a single candidate output patch is selected, then the first output patch is the selected candidate output patch, otherwise the first output patch is derived using the selected candidate output patches.

[0163] C2. The apparatus of embodiment Cl, wherein the apparatus is further configured to perform the method of any one of embodiments A2-A14.

[0164] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0165] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method (1900) for image processing, the method comprising: obtaining (si 902) a first input patch (611) comprising a first set of one or more processed samples within a first region of an image; processing (sl904) the first input patch using a first part (601) of a neural networkbased, NN-based, filter to produce a first set of NN outputs (613); obtaining (si 906) a first output patch (615) based on a determined type of the first input patch, the first output patch comprising a second set of one or more processed samples within the first region of the image, wherein obtaining the first output patch based on the determined type of the first input patch comprises:1) based on the determined type, selecting at least one NN from a set of two or moreNNs and producing the first output patch using the selected NN(s) and the first set of NN outputs (613), where the set of two or more NNs is part of a second part (602) of the NN based filter,2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs (613), where the set of two or more NN layers is part of a second part (602) of the NN based filter,3) based on the determined type, selecting at least one set of weights and producing the first output patch using the selected set of weights, at least one NN, and the first set of NN outputs (613), or4) using a set of two or more NNs and the first set of NN outputs (613) to produce two or more candidate output patches and selecting, based on the determined type, at least one of the candidate output patches, wherein, if a single candidate output patch is selected, then the first output patch is the selected candidate output patch, otherwise the first output patch is derived using the selected candidate output patches.

2. The method of claim 1, wherein the NN-based filter is an NN-based loop filter.

3. The method of claim 1 or 2, wherein the first part of the NN-based filter comprises a plurality of convolutional layers, and the first set of NN outputs comprises a plurality of channels.

4. The method of any one of claims 1-3, wherein obtaining the first output patch based on the determined type of the first input patch comprises:1) based on the determined type, selecting at least one NN from a set of two or moreNNs and producing the first output patch using the selected NN(s) and the first set of NN outputs (613), where the set of two or more NNs is part of a second part (602) of the NN based filter, or2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs (613), where the set of two or more NN layers is part of a second part (602) of the NN based filter, and the second part of the NN-based filter obtains a type value indicating the type of the first input patch.

5. The method of claim 4, wherein the first part of the NN-based filter provides the obtained type value to the second part of the NN-based filter.

6. The method of claim 4, wherein the obtained type value is inferred from one or more encoded parameters from a bitstream.

7. The method of any one of claims 4-6, wherein obtaining the first output patch comprises selecting at least one NN from the set of two or more NNs, each NN included in the set of two or more NNs is associated with one or more type values, and selecting at least one NN from the set of two or more NNs comprises selecting a first NN from the set of two or more NNs as a result of determining that the first NN is associated with a type value that matches the obtained type value.

8. The method of claim 7, wherein each NN included in the set of two or more NNs is a convolutional NN.

9. The method of claim 8, wherein each convolutional NN is trained to remove artifacts.

10. The method of any one of claims 7-9, wherein the obtained type value is a quantization parameter, QP, value associated with the first input patch.

11. The method of any one of claims 7-9, wherein the obtained type value is, or is based on, a ratio of a lowest frequency component in the first input patch to a second lowest frequency component in the first input patch.

12. The method of claim 11, wherein the obtained type value is type A if the ratio is less than a threshold, or the obtained type value is type B if the ratio is greater than the threshold.

13. The method of claim 12, wherein the first part of the NN-based filter performs an affine transform on the input first patch, and the first part selects type A as the obtained type value if the ratio is less than the threshold or less than or equal to the threshold, otherwise the first part selects type B as the obtained type value.

14. The method of any one of claims 7-13, wherein obtaining the first output patch comprises selecting at least a first NN and a second NN from the set of two or more NNs, the first NN uses the first set of NN outputs to produce a first intermediate patch, the second NN uses the first set of NN outputs to produce a second intermediate patch, and obtaining the output patch comprises blending the first and the second intermediate patches.

15. The method of any one of claims 1-14, wherein the method further comprises, prior to obtaining the first output patch based on the determined type of the first input patch, determining the type of the first input patch.

16. The method of claim 15, wherein determining the type of the first input comprises obtaining a type value indicating the type of the first input patch, or determining the type of the first input comprises obtaining a type value indicating the type of the first input patch.

17. A computer program (2043) comprising instructions (2044) which when executed by processing circuitry (2002) cause the processing circuitry (2002) to perform the method of any one of the above claims.

18. A carrier containing the computer program of claim 17, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (2042).

19. An apparatus (2000) for image processing, the apparatus being configured to perform a method comprising: obtaining (si 902) a first input patch (611) comprising a first set of one or more processed samples within a first region of an image; processing (sl904) the first input patch using a first part (601) of a neural networkbased, NN-based, filter to produce a first set of NN outputs (613); obtaining (si 906) a first output patch (615) based on a determined type of the first input patch, the first output patch comprising a second set of one or more processed samples within the first region of the image, wherein obtaining the first output patch based on the determined type of the first input patch comprises:1) based on the determined type, selecting at least one NN from a set of two or more NNs and producing the first output patch using the selected NN(s) and the first setof NN outputs (613), where the set of two or more NNs is part of a second part (602) of the NN based filter,2) based on the determined type, selecting at least one NN layer from a set of two or more NN layers and producing the first output patch using the selected NN layer(s) and the first set of NN outputs (613), where the set of two or more NN layers is part of a second part (602) of the NN based filter,3) based on the determined type, selecting at least one set of weights and producing the first output patch using the selected set of weights, at least one NN, and the first set of NN outputs (613), or4) using a set of two or more NNs and the first set of NN outputs (613) to produce two or more candidate output patches and selecting, based on the determined type, at least one of the candidate output patches, wherein, if a single candidate output patch is selected, then the first output patch is the selected candidate output patch, otherwise the first output patch is derived using the selected candidate output patches.

20. The apparatus of claim 19, wherein the apparatus is further configured to perform the method of any one of claims 2-16.

Images