Apparatus and method employing neural network layer(s) for distributed processing

Abstract

An example method includes: dividing an input into two or more portions and / or generating two or more copies of the input with a demultiplexor of a distributed processing unit; differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input with two or more processing branches of the distributed processing unit to generate respective intermediate outputs; and combining the respective intermediate outputs with a combiner of the distributed processing unit to obtain an output or data from which the output is derivable, wherein at least one or more neural network layers are configured to serve at least as the two or more processing branches.

Description

APPARATUS AND METHOD EMPLOYING NEURAL NETWORK LAYER(S) FOR DISTRIBUTEDPROCESSINGTECHNICAL FIELD

[0001] An apparatus, method and computer program product are provided that utilize two or more processing branches embodied as neural network layer(s) for use, for example, in conjunction with an encoder and / or a decoder.BACKGROUND

[0002] A video codec consists of an encoder that transforms the input video into a compressed representation suited for storage and / or transmission and a decoder that can decompress the compressed video representation back to a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form, that is, at lower bitrate.

[0003] Typical hybrid video codecs, for example, video codecs in compliance with ITU-T H.263 and H.264, encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted, such as by motion compensation or by spatial means. Motion compensation involves finding and indicating an area in one or more previously coded video frames that corresponds closely to the block being coded, while the spatial technique uses the pixel values proximate the block to be coded in a specified manner. Secondly, the prediction error, e.g. the difference between the predicted block of pixels and the original block of pixels, is coded. This coding is typically done by transforming the difference in pixel values using a specified transform, e.g., Discrete Cosine Transform (DCT) or a variant of DCT, quantizing the resulting coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation, e.g., picture quality, and the size of the resulting coded video representation, e.g., file size or transmission bitrate.

[0004] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction, the sources of prediction are previously decoded pictures, namely, reference pictures.

[0005] In temporal inter prediction, the sources of prediction are previously decoded pictures in the same scalable layer. In intra block copy (IBC), also known as intra-block-copy prediction, prediction may be applied similarly to temporal inter prediction, but the reference picture is the current picture and only previously decoded samples can be referenced in the prediction process. Inter-layer or inter- view prediction may be applied similarly to temporal inter prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal inter prediction only, while in other cases inter prediction may refercollectively to temporal inter prediction and any of intra block copy, inter-layer prediction, and interview prediction provided that they are performed with the same or similar process than temporal prediction. Inter prediction, temporal inter prediction, or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0006] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in a spatial or transform domain such that either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0007] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0008] The decoder reconstructs the output video by applying a prediction technique similar to the encoder to form a predicted representation of the pixel blocks, such as by using the motion or spatial information created by the encoder and stored in the compressed representation, and to decode the prediction error, such as by performing the inverse operation of the prediction error coding to recover the quantized prediction error signal in the spatial pixel domain. After applying prediction and prediction error decoding, the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering to improve the quality of the output video before passing the output video for display and / or storing the output video as prediction reference for the forthcoming frames in the video sequence.

[0009] In typical video codecs, the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (on the encoder side) or decoded (on the decoder side) and the prediction source block in one of the previously coded or decoded pictures. In order to represent motion vectors efficiently, the motion vectors are typically coded differentially with respect to block specific predicted motion vectors. In some video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Alternatively, motion vector predictions may be created by generating a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signalling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of a previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or or co-located blocks in a temporal reference picture. Moreover, typical high efficiency video codecs employ an additionalmotion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes a motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signalled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0010] In typical video codecs, the prediction residual after motion compensation is first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform that can, in many cases, help reduce this correlation and provide more efficient coding.

[0011] Some video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g. the desired macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor X to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:C = D + R where C is the Lagrangian cost to be minimized, D is the image distortion, e.g., mean squared error, with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0012] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or the like. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units. A prefix SEI NAL unit can start a picture unit or the like and a suffix SEI NAL unit can end a picture unit or the like. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, postprocessing of decoded pictures, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in the H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards, and the user data SEI messages enable entities to specify SEI messages for their own use. The standards may contain the syntax and semantics for the specified SEI messages, but a process for handling the messages by the recipient may not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when the encoders create SEI message^), but decoders may not be required to process SEI messages for output order conformance. By including the syntax and semantics of SEI messages in standards allows different system specifications to interpret the supplemental information identically and hence interoperate. It is intended thatsystem specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.

[0013] While encoders and decoders are effective, their encoding and decoding performance and, in turn, the resulting video storage and / or transmission experience may be still be desirably improved.BRIEF SUMMARY

[0014] According to some aspects, there is provided the subject matter of the independent claims. Some further aspects are defined in the dependent claims. The embodiments that do not fall under the scope of the claims are to be interpreted as examples useful for understanding the disclosure.

[0015] In an example embodiment, an apparatus is provided that includes at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to divide an input into two or more portions and / or generate two or more copies of the input, differently process respective copies of the two or more copies and / or portions of the two or more portions of the input to generate respective intermediate outputs and combine the respective intermediate outputs to obtain an output or data from which the output is derivable. The instructions, when executed by the at least one processor, generate a distributed processing unit comprising a demultiplexer configured to divide the input into the two or more portions and / or generate the two or more copies of the input, two or more processing branches configured to differently process the respective copies of the two or more copies and / or portions of the two or more portions of the input and a combiner configured to combine the respective intermediate outputs. The instructions, when executed by the at least one processor, define at least one or more neural network layers that are configured to serve at least as the two or more processing branches.

[0016] The instructions, when executed by the at least one processor in accordance with an example embodiment, define a neural network comprising the at least one or more neural network layers that are configured to serve at least as the two or more processing branches as well as one or more previous neural network layers that are configured to generate the input to the distributed processing unit. In another embodiment, the instructions, when executed by the at least one processor, define a neural network comprising the at least one or more neural network layers that are configured to serve at least as the two or more processing branches as well as one or more following neural network layers that are configured to process the output of the distributed processing unit.

[0017] The instructions, when executed by the at least one processor, further cause the apparatus of an example embodiment to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input by performing a modulation operation to multiply by one or more multiplier parameters at least one copy of the two or more copies and / or portion of the two or more portions of the input. In another embodiment, the instructions, when executed bythe at least one processor, further cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input by performing a modulation operation to add one or more additive parameters to at least one copy of the two or more copies and / or portion of the two or more portions of the input. The instructions, when executed by the at least one processor, may define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit by defining a convolutional layer that has a kernel size of 1x1 and that comprises at least a portion of at least one of the two or more processing branches. In an example embodiment, the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit by defining a grouped convolutional layer that is configured to apply different kernels to different groups of channels of the input and that comprises at least a portion of at least one of the two or more processing branches.

[0018] The instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least two of the two or more processing branches are configured in accordance with an example embodiment to process respective copies of the two or more copies and / or portions of the two or more portions of the input in accordance with different resolutions and / or different directions. The instructions, when executed by the at least one processor, further cause the apparatus of an example embodiment to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input by performing an identity operation. The instructions, when executed by the at least one processor in accordance with an example embodiment, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different resolutions and / or different directions. In one embodiment, the data processed by the at least two processing sub-branches is the same. In another embodiment, the data processed by one of the at least two processing sub-branches is different than the data processed by another of the at least two processing sub-branches. In an embodiment, the different directions comprise a vertical direction and a horizontal direction.

[0019] The instructions, when executed by the at least one processor in accordance with an example embodiment, define one or more neural network layers that are configured to serve as a distributed processing unit that is configured to process the input at a resolution different than the resolution at which the input is provided to the distributed processing unit. In one embodiment, the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least two of the two or more processing branches are configured to process the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions. Theinstructions, when executed by the at least one processor in accordance with an example embodiment, define one or more neural network layers that are configured to serve as a distributed processing unit that is comprised of a plurality of components configured to operate at different resolutions. The instructions, when executed by the at least one processor in accordance with an example embodiment, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least two of the two or more processing branches are configured to process the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions. In an embodiment, the different directions comprise a vertical direction and a horizontal direction.

[0020] The instructions, when executed by the at least one processor in accordance with an example embodiment, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that any processing branch is configured to generate an intermediate output comprising a first subset of data and a second subset of data. The first and second subsets of data are disjoint subsets of data. The first subset of data generated by any processing branch corresponds to or is associated with a first portion of an output of the distributed processing unit. The second subset of data generated by a particular processing branch corresponds to or is associated with a particular portion of an output of the distributed processing unit. The instructions, when executed by the at least one processor, define the combiner of the distributed processing unit of this embodiment to comprise a first combiner that is configured to combine the first subsets of data generated by the two or more processing branches.

[0021] The combiner of this embodiment may further comprise a second combiner that is configured to combine the first output and the second subsets of data generated by the two or more processing branches to form a second output from which the output of the distributed processing unit is derived. In an example embodiment, the combiner further comprises a second combiner configured to combine the second subsets of data generated by the two or more processing branches to form a second output and a third combiner configured to combine the first output and the second output to form a third output from which the output of the distributed processing unit is derived. In an example embodiment, the second processing branch is defined by one or more neural network layers prior to which the respective copy of the two or more copies and / or portion of the two or more portions of the input is downsampled and after which an output of the one or more neural network layers is upsampled. In another embodiment, the second processing branch that is configured to process a respective copy of the two or more copies and / or portion of the two or more portions of the input comprised of a plurality of channels comprises a first sub-branch configured to process a subset of the plurality of channels that are associated with one another and a second sub-branch configured to process a subset of the plurality of channels that are disjoint. The first and second sub-branches are configured to comprise different neural network layers.

[0022] The input of an example embodiment is a tensor. The demultiplexer of an example embodiment is configured to divide the input into the two or more portions and / or generate the two or more copies of the input based on one or more dimensions of the input. In another embodiment, the demultiplexer is configured to divide the input into the two or more portions and / or generate the two or more copies of the input based on channels, spatial location or temporal location of the input. In an example embodiment, the two or more copies or the two or more portions into which the input is divided comprise respective two or more subsets of channels, and at least two of the two or more subsets of channels are either disjoint or overlapping. The combiner of an example embodiment is configured to combine the respective intermediate outputs by combining two or more tensors by one or more of concatenation, summation or stacking.

[0023] In another embodiment, a method is provided that includes generating two or more copies of an input and / or dividing the input into two or more portions with a demultiplexor of a distributed processing unit, differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input with two or more processing branches of the distributed processing unit to generate respective intermediate outputs and combining the respective intermediate outputs with a combiner of the distributed processing unit to obtain an output or data from which the output is derivable. At least one or more neural network layers are configured to serve at least as the two or more processing branches.

[0024] The method of an example embodiment further comprises generating the input to the distributed processing unit with one or more previous neural network layers. In this embodiment, the at least one or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more previous neural network layers. The method of another embodiment further comprises processing the output of the distributed processing unit with one or more following neural network layers. In this embodiment, the at least one or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more following neural network layers.

[0025] Differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises, in one example embodiment, performing a modulation operation to multiply by one or more multiplier parameters at least one portion of the two or more portions of the input. In an example embodiment, differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises performing a modulation operation to add one or more additive parameters to at least one portion of the two or more portions of the input. The one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit in an example embodiment define a convolutional layer that has a kernel size of 1x1 and that comprises at least a portion of at least one of the two or more processing branches. In another example embodiment, the one or more neural networklayers that are configured to serve as the two or more processing branches of the distributed processing unit define a grouped convolutional layer that is configured to apply different kernels to different groups of channels of the input and that comprises at least a portion of at least one of the two or more processing branches.

[0026] Differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input may include processing respective copies of the two or more copies and / or portions of the two or more portions of the input in accordance with different resolutions and / or different directions. In an example embodiment, differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises performing an identity operation. In one embodiment, at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different resolutions. In another embodiment, the at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different directions. The data processed by the at least two processing sub-branches may be the same. In another embodiment, the data processed by one of the at least two processing sub-branches is different than the data processed by another of the at least two processing sub-branches. In an embodiment, the different directions comprise a vertical direction and a horizontal direction.

[0027] Differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input may include processing the input at a resolution different than the resolution at which the input is provided to the distributed processing unit. In an example embodiment, differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises processing the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions. In one embodiment, the one or more neural network layers are configured to serve as the distributed processing unit that is comprised of a plurality of components configured to operate at different resolutions. Differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input may include processing the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions with at least two of the two or more processing branches. In an embodiment, the different directions comprise a vertical direction and a horizontal direction.

[0028] Differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises, in accordance with an example embodiment, generating an intermediate output comprising a first subset of data and a second subset of data with any processing branch. The first and second subsets of data are disjoint subsets of data. The first subset of data generated by any processing branch corresponds to or is associated with a first portion of an output of the distributed processing unit. The second subset of data generated by a particular processingbranch corresponds to or is associated with a particular portion of an output of the distributed processing unit. Combining the respective intermediate outputs comprises combining the first subsets of data generated by the two or more processing branches.

[0029] The method of this example embodiment may further comprise combining the first output and the second subsets of data generated by the two or more processing branches to form a second output from which the output of the distributed processing unit is derived. In an example embodiment, the method further includes combining the second subsets of data generated by the two or more processing branches to form a second output and combining the first output and the second output to form a third output from which the output of the distributed processing unit is derived. The method of an example embodiment also includes downsampling the respective copy of the two or more copies and / or portion of the two or more portions of the input prior to the second processing branch and up- sampling an output of the one or more neural network layers following the second processing branch. In an example embodiment, processing, with the second processing branch, a respective copy of the two more copies and / or portion of the two or more portions of the input comprised of a plurality of channels comprises processing a subset of the plurality of channels that are associated with one another with a first sub-branch and processing a subset of the plurality of channels that are disjoint with a second sub-branch, and the first and second sub-branches are configured to comprise different neural network layers.

[0030] The input of an example embodiment is a tensor. In one embodiment, generating the two or more copies of the input and / or dividing the input into the two or more portions comprises generating the two or more copies of the input and / or dividing the input into the two or more portions based on one or more dimensions of the input and / or based on channels, spatial location and / or temporal location of the input. In an example embodiment, the two or more copies or the two or more portions into which the input is divided comprise respective two or more subsets of channels, and at least two of the two or more subsets of channels are either disjoint or overlapping. In one embodiment, combining the respective intermediate outputs comprises combining two or more tensors by one or more of concatenation, summation or stacking.

[0031] In a further embodiment, a computer program product is provided that includes anon- transitory computer-readable storage medium including instructions that, when executed by an apparatus, cause the apparatus to serve as a demultiplexor of a distributed processing unit to divide an input into two or more portions and / or generate the two or more copies, to serve as two or more processing branches of the distributed processing unit to differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input to generate respective intermediate outputs and to serve as a combiner of the distributed processing unit to combine the respective intermediate outputs to obtain an output or data from which the output is derivable. Theinstructions that define at least one or more neural network layers that serve at least as the two or more processing branches.

[0032] The instructions of an example embodiment also cause the apparatus to generate the input to the distributed processing unit with one or more previous neural network layers. In this embodiment, the at least one or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more previous neural network layers. The instructions of another embodiment also cause the apparatus to process the output of the distributed processing unit with one or more following neural network layers. In this embodiment, the at least one or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more following neural network layers.

[0033] The instructions that cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input comprise, in one example embodiment, instructions that cause the apparatus to perform a modulation operation to multiply by one or more multiplier parameters at least one portion of the two or more portions of the input. In an example embodiment, the instructions that cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input comprise instructions that cause the apparatus to perform a modulation operation to add one or more additive parameters to at least one portion of the two or more portions of the input. The one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit in an example embodiment define a convolutional layer that has a kernel size of 1x1 and that comprises at least a portion of at least one of the two or more processing branches. In another example embodiment, the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit define a grouped convolutional layer that is configured to apply different kernels to different groups of channels of the input and that comprises at least a portion of at least one of the two or more processing branches.

[0034] The instructions that cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input may include instructions that cause the apparatus to process respective copies of the two or more copies and / or portions of the two or more portions of the input in accordance with different resolutions and / or different directions. In an example embodiment, the instructions that cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input comprise instructions that cause the apparatus to perform an identity operation. In one embodiment, at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different resolutions. In another embodiment, the at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different directions. The data processed by the at leasttwo processing sub-branches may be the same. In another embodiment, the data processed by one of the at least two processing sub-branches is different than the data processed by another of the at least two processing sub-branches. In an embodiment, the different directions comprise a vertical direction and a horizontal direction.

[0035] The instructions that cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input may include instructions that cause the apparatus to process the input at a resolution different than the resolution at which the input is provided to the distributed processing unit. In an example embodiment, the instructions that cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input comprise instructions that cause the apparatus to process the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions. In one embodiment, the one or more neural network layers are configured to serve as the distributed processing unit that is comprised of a plurality of components configured to operate at different resolutions. The instructions that cause the apparatus to differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input may include instructions that cause the apparatus to process the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions with at least two of the two or more processing branches. In an embodiment, the different directions comprise a vertical direction and a horizontal direction.

[0036] The instructions that cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input comprise, in accordance with an example embodiment, instructions that cause the apparatus to generate an intermediate output comprising a first subset of data and a second subset of data with any processing branch. The first and second subsets of data are disjoint subsets of data. The first subset of data generated by any processing branch corresponds to or is associated with a first portion of an output of the distributed processing unit. The second subset of data generated by a particular processing branch corresponds to or is associated with a particular portion of an output of the distributed processing unit. The instructions that cause the apparatus to combine the respective intermediate outputs comprise instructions that cause the apparatus to combine the first subsets of data generated by the two or more processing branches.

[0037] The instructions of this example embodiment may further comprise instructions that cause the apparatus to combine the first output and the second subsets of data generated by the two or more processing branches to form a second output from which the output of the distributed processing unit is derived. In an example embodiment, the instructions further include instructions that cause the apparatus to combine the second subsets of data generated by the two or more processing branches to form a second output and to combine the first output and the second output to form a third outputfrom which the output of the distributed processing unit is derived. The instructions of an example embodiment also include instructions that cause the apparatus to downsample the respective copy of the two or more copies and / or portion of the two or more portions of the input prior to the second processing branch and to upsample an output of the one or more neural network layers following the second processing branch. In an example embodiment, the instructions that cause the apparatus to process, with the second processing branch, a respective copy of the two or more copies and / or portion of the two or more portions of the input comprised of a plurality of channels comprise instructions that cause the apparatus to process a subset of the plurality of channels that are associated with one another with a first sub-branch and to process a subset of the plurality of channels that are disjoint with a second sub-branch, and the first and second sub-branches are configured to comprise different neural network layers.

[0038] The input of an example embodiment is a tensor. In one embodiment, the instructions that cause the apparatus to divide the input into the two or more portions and / or generate the two or more copies of the input comprise instructions that cause the apparatus to divide the input into the two or more portions and / or generate the two or more copies of the input based on one or more dimensions of the input and / or based on channels, spatial location and / or temporal location of the input. In an example embodiment, the two or more copies or the two or more portions into which the input is divided comprise respective two or more subsets of channels, and at least two of the two or more subsets of channels are either disjoint or overlapping. In one embodiment, the instructions that cause the apparatus to combine the respective intermediate outputs comprise instructions that cause the apparatus to combine two or more tensors by one or more of concatenation, summation or stacking.

[0039] In yet another embodiment, an apparatus is provided that includes means, such as a demultiplexor of a distributed processing unit, for generating the two or more copies of the input and / or dividing an input into two or more portions, means, such as two or more processing branches of the distributed processing unit, for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input to generate respective intermediate outputs and means, such as a combiner of the distributed processing unit, for combining the respective intermediate outputs to obtain an output or data from which the output is derivable. The means for combining include at least one or more neural network layers that are configured to serve at least as the two or more processing branches.

[0040] The apparatus of an example embodiment further comprises means for generating the input to the distributed processing unit with one or more previous neural network layers. In this embodiment, the at least one or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more previous neural network layers. The apparatus of another embodiment further comprises means for processing the output of the distributed processing unit with one or more following neural network layers. In this embodiment, the at leastone or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more following neural network layers.

[0041] The means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprise, in one example embodiment, means for performing a modulation operation to multiply by one or more multiplier parameters at least one portion of the two or more portions of the input. In an example embodiment, the means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprise means for performing a modulation operation to add one or more additive parameters to at least one portion of the two or more portions of the input. The one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit in an example embodiment define a convolutional layer that has a kernel size of 1x1 and that comprises at least a portion of at least one of the two or more processing branches. In another example embodiment, the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit define a grouped convolutional layer that is configured to apply different kernels to different groups of channels of the input and that comprises at least a portion of at least one of the two or more processing branches.

[0042] The means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input may include means for processing respective copies of the two or more copies and / or portions of the of the two or more portions input in accordance with different resolutions and / or different directions. In an example embodiment, the means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprise means for performing an identity operation. In one embodiment, at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different resolutions. In another embodiment, the at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different directions. The data processed by the at least two processing sub-branches may be the same. In another embodiment, the data processed by one of the at least two processing sub-branches is different than the data processed by another of the at least two processing sub-branches. In an embodiment, the different directions comprise a vertical direction and a horizontal direction.

[0043] The means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input may include means for processing the input at a resolution different than the resolution at which the input is provided to the distributed processing unit. In an example embodiment, the means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprise means for processing the respective copies of the two or more copies and / or portions of the two or more portions of theinput data in accordance with different directions. In one embodiment, the one or more neural network layers are configured to serve as the distributed processing unit that is comprised of a plurality of components configured to operate at different resolutions. The means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input may include means for processing the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions with at least two of the two or more processing branches.

[0044] The means for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprise, in accordance with an example embodiment, means for generating an intermediate output comprising a first subset of data and a second subset of data with any processing branch. The first and second subsets of data are disjoint subsets of data. The first subset of data generated by any processing branch corresponds to or is associated with a first portion of an output of the distributed processing unit. The second subset of data generated by a particular processing branch corresponds to or is associated with a particular portion of an output of the distributed processing unit. The means for combining the respective intermediate outputs comprise means for combining the first subsets of data generated by the two or more processing branches.

[0045] The apparatus of this example embodiment may further comprise means for combining the first output and the second subsets of data generated by the two or more processing branches to form a second output from which the output of the distributed processing unit is derived. In an example embodiment, the apparatus further includes means for combining the second subsets of data generated by the two or more processing branches to form a second output and combining the first output and the second output to form a third output from which the output of the distributed processing unit is derived. The apparatus of an example embodiment also includes means for downsampling the respective copy of the two or more copies and / or portion of the two or more portions of the input prior to the second processing branch and means for upsampling an output of the one or more neural network layers following the second processing branch. In an example embodiment, the means for processing, with the second processing branch, a respective copy of the two or more copies and / or portion of the two or more portions of the input comprised of a plurality of channels comprises means for processing a subset of the plurality of channels that are associated with one another with a first sub-branch and means for processing a subset of the plurality of channels that are disjoint with a second sub-branch, and the first and second sub-branches are configured to comprise different neural network layers.

[0046] The input of an example embodiment is a tensor. In one embodiment, the means for generating the two or more copies of the input and / or dividing the input into the two or more portions comprises means for generating the two or more copies of the input and / or dividing the input into the two or more portions based on one or more dimensions of the input and / or based on channels, spatiallocation and / or temporal location of the input. In an example embodiment, the two or more copies or the two or more portions into which the input is divided comprise respective two or more subsets of channels, and at least two of the two or more subsets of channels are either disjoint or overlapping. In one embodiment, the means for combining the respective intermediate outputs comprise means for combining two or more tensors by one or more of concatenation, summation or stacking.LIST OF THE DRAWINGS

[0047] In the following, the embodiments of the present disclosure will be described in greater detail with reference to the embodiments and the accompanying drawings, in which:

[0048] Figure 1 illustrates a video coding pipeline;

[0049] Figure 2 illustrates an end-to-end learned codec;

[0050] Figure 3 illustrates a system pipeline for video coding for machines:

[0051] Figure 4 illustrates encoder-side operations for an overfitting process;

[0052] Figure 5 illustrates decoder or receiver-side operations for an overfitting process;

[0053] Figure 6 illustrates a neural network based in-loop filter;

[0054] Figure 7 illustrates an apparatus that is capable of being configured in accordance with an example embodiment;

[0055] Figure 8 illustrates the operations performed, such as by the apparatus of Figure 7, in accordance with an example embodiment;

[0056] Figure 9 illustrates a processing unit in accordance with an example embodiment;

[0057] Figure 10 illustrates a processing unit in accordance with another example embodiment;

[0058] Figure 11 illustrates a processing unit in which the demultiplexor is configured to divide the input into two disjoint subsets of channels in accordance with an example embodiment;

[0059] Figure 12 illustrates a distributed processing unit having two processing branches with a first processing branch having a convolution layer and a second processing branch having a modulation operation based on multiplier parameters in accordance with an example embodiment;

[0060] Figure 13 illustrates a processing unit having a third processing branch that is a skip connection in accordance with an example embodiment;

[0061] Figure 14 illustrates a processing unit comprised in a neural network based in-loop filter in accordance with an example embodiment;

[0062] Figure 15-17 illustrate distributed processing units in accordance with example embodiments;

[0063] Figure 18 illustrates a processing unit in accordance with an example embodiment;

[0064] Figure 19 illustrates a distributed processing unit in accordance with an example embodiment;

[0065] Figure 20 illustrates a processing unit in accordance with another example embodiment;

[0066] Figures 21 and 22 illustrate distributed processing units in accordance with example embodiments;

[0067] Figure 23 illustrates a processing unit in accordance with an example embodiment;

[0068] Figure 24 illustrates a distributed processing unit in accordance with an example embodiment;

[0069] Figures 25-27 illustrate processing units in accordance with other example embodiments;

[0070] Figure 28 illustrates a distributed processing unit in accordance with an example embodiment;

[0071] Figure 29 illustrates a processing unit in accordance with an example embodiment;

[0072] Figure 30 illustrates a distributed processing unit in accordance with another example embodiment;

[0073] Figure 31 illustrates a processing unit in accordance with another example embodiment;

[0074] Figure 32 illustrates a distributed processing unit in accordance with a further example embodiment;

[0075] Figures 33-43 illustrate processing units in accordance with other example embodiments;

[0076] Figures 44 and 45 illustrate distributed processing units in accordance with other example embodiments; and

[0077] Figure 46-49 illustrate processing units in accordance with further example embodiments.DESCRIPTION OF EMBODIMENTS

[0078] The following embodiments are exemplary. Although the specification may refer to “an”, “one”, or “some” embodiment(s) in several locations of the text, this does not necessarily mean that each reference is made to the same embodiment(s), or that a particular feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments. Further, when a particular feature, structure, or characteristic is described in connection of an embodiment, it is within the knowledge of one skilled in the art to apply such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. It shall be understood that although the terms “first,” “second” and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0079] For the purposes of the present disclosure, the phrases “at least one of A or B”, “at least one of A and B”, and “A and / or B” means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0080] Neural networks (NNs) may be used in the context of image and video compression in order to improve the encoding and decoding performance. A neural network may be described as acomputation graph consisting of several layers of computation. Each layer may consist of one or more units with each unit performing an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, e.g., values which can be learned from training data. There may be other learnable parameters, such as those of batchnormalization layers. As used herein, the terms “model”, “neural network”, “neural net” and “network” are interchangeable. In addition, the weights of neural networks are sometimes referred to as learnable parameters or simply as parameters.

[0081] In some neural networks, such as convolutional neural networks for image classification, initial layers close to the input data extract semantically low-level features such as edges and textures in images, whereas intermediate layers extract more high-level features. After the feature extraction layers, there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc.

[0082] Neural networks are being utilized in an ever-increasing number of applications for many different types of devices, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc.

[0083] One property of neural networks and other machine learning tools is that they are able to learn properties from input data, such as in a supervised manner or in an unsupervised manner. This learning is a result of a training algorithm, or of a meta-level neural network that provides the training signal.

[0084] In general, the training algorithm consists of changing some properties of the neural network so that the output of the neural network is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class or category index which indicates the class or category to which the object in the input image belongs. Training may be performed to minimize or decrease the error of the neural network’s output, also referred to as the loss or loss function. Examples of losses are mean squared error, crossentropy, etc. In recent deep learning techniques, training is an iterative process. At each iteration, the algorithm modifies the weights of the neural network to make a gradual improvement of the network’ s output, such as by gradually decreasing the loss, by means of gradient descent technique. In one example, at each training iteration, gradients of the loss function with respect to one or more weights or parameters of the neural network are computed, such as by a backpropagation technique; the computed gradients are then used by an optimization routine, such as Adam or Stochastic Gradient Descent (SGD), to obtain an update to the one or more weights or parameters.

[0085] Training a neural network is an optimization process, but the final goal may be different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn theproperties of the data distribution from a limited training dataset. In other words, the goal is to learn from a limited training dataset in order to generalize to previously unseen data, that is, data which was not used for training the model. This process is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, such as to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to determine if the network is learning. If the neural network is learning, the training set error should decrease. Otherwise, the neural network is in the regime of underfitting. The errors are also monitored during the training process to determine if the network is learning to generalize. If the neural network is learning to generalize, the validation set error decreases and is not much higher than the training set error. However, if the training set error is low, but the validation set error is much higher than the training set error, or the validation set error does not decrease, or the validation set error even increases, the neural network may be in the regime of overfitting. The regime of overfitting means that the neural network has just memorized the properties of the training set and performs well only on that set, but performs poorly on a set not used for tuning its parameters.

[0086] With respect to neural networks used in the context of image and video compression, two approaches have generally been taken. In a first approach, neural networks are used to replace one or more of the components of a traditional codec, such as a VVC / H.266-compliant codec. Traditional codecs are codecs having components and parameters that are not learned or trained from data by means of machine learning techniques. Examples of components that may be implemented as neural networks are: (i) an in-loop filter, such as a neural network that serves as an additional in-loop filter with respect to the traditional loop filters, or a network that works as the only additional in-loop filter, thus replacing any other in-loop filter, (ii) a neural network that performs intra-frame prediction, (iii) a neural network that performs inter-frame prediction, (iv) a neural network that performs a transform and / or an inverse transform, (v) a neural network that provides a probability model for lossless coding, as well as others.

[0087] In a second approach, commonly referred to as “end-to-end learned compression” (or an end-to-end learned codec), neural networks are used as the main components of an image / video codec. However, the codec may still include components which are not based on machine learning techniques. In this second approach, two design options are implemented. In a first option, a video coding pipeline is utilized, but most or all the components are replaced with neural networks as shown in Figure 1. In this example, the forward and inverse transforms are replaced with two neural networks. The loop filter and potentially other components may also be formed by a neural network.

[0088] In a second option (also referred to as end-to-end learned coding), the entire pipeline may be redesigned as a neural network auto-encoder with a quantization and lossless coding in an intermediate portion. The neural network auto-encoder may include an encoder neural network (also referred to as neural network based encoder, or NN encoder) configured to perform a non-linear transformation of the input to generate an output that is typically referred to as latent tensor. The neural network auto-encoder may also be configured to perform quantization and lossless encoding of the output of the encoder neural network, as well as lossless decoding and dequantization. The neural network auto-encoder may also include a decoder neural network (also referred to as neural network based decoder, or NN decoder) configured to perform a non-linear inverse transformation from a dequantized latent tensor to a reconstructed input. Even in end-to-end learned approaches, there may be components which are not learned from data, such as an arithmetic codec.

[0089] With respect to this second option, Figure 2 illustrates an example of neural networkbased end-to-end learned coding, such as an end-to-end learned video coding system or an end-to-end learned image coding system. Even though some examples are provided with respect to coding images or videos, it is to be understood that other types of data may be coded in a similar way, such as audio, speech, text, features, etc. As shown in Figure 2, however, a typical neural network-based end- to-end learned coding system contains an encoder and a decoder.

[0090] The encoder 30 of Figure 2 includes an encoder NN 31, a quantizer or quantization 32, a probability model 33, and a lossless encoder 34, such as an arithmetic encoder. The decoder 35 includes a lossless decoder 36, such as an arithmetic decoder, a probability model 37, a dequantizer or dequantization 38 and a decoder NN 39. The probability model at encoder side and the probability model at decoder side may be same or substantially the same. For example, they may be two instances of the probability model may be copies of the same probability model. The lossless encoder and the lossless decoder form a lossless codec 40. A lossless codec may be an entropy-based lossless codec. An example of lossless codec is an arithmetic codec, such as a context-adaptive binary arithmetic coding (CABAC). The encoder NN and decoder NN are typically two neural networks, or may be mainly comprised of neural network components.

[0091] The probability model 33, 37 may also be a neural network and / or may mainly include neural network components. The probability model may be referred to as a neural network based probability model or learned probability model. A lossless codec 40 may refer to a system that also includes the probability model, in addition to, for example, an arithmetic encoder 34 and an arithmetic decoder 36. In some embodiments, the quantizer 324, dequantizer 38 and lossless codec 40 are not based on neural network components. However, in other embodiments, the quantizer, dequantizer and / or lossless codec may also be formed of neural network components.

[0092] In the embodiment of Figure 2, the encoder NN 31 receives an input x, which may include, for example, an image to be compressed. The encoder NN outputs a latent tensor z. In oneexample, the latent tensor may be a three dimensional (3D) tensor, in which the three dimensions of a tensor represents a channel dimension, a vertical dimension (also sometimes referred to as a height dimension) and a horizontal dimension (also sometimes referred to as a width dimension). In another example, the latent tensor may be a four dimensional (4D) tensor in which the four dimensions of a tensor represent a sample dimension (also sometimes referred to as batch dimension, which is the dimension along which different samples of data can be placed), a channel dimension, a vertical dimension (also sometimes referred to as height dimension) and a horizontal dimension (also sometimes referred to as width dimension). The latent tensor is input to a quantization operation 32 which produces a quantized latent tensor zq. The quantized latent tensor is lossless-encoded into a bitstream b by the lossless encoder 34, based also on the output of the probability model 33. In particular, the probability model takes as input at least part of the quantized latent tensor and outputs an estimate of a probability or an estimate of a probability distribution or an estimate of one or more parameters of a probability distribution for one or more elements of the quantized latent tensor. The bitstream represents an encoded or compressed version of the input x.

[0093] The bitstream is lossless-decoded by the lossless decoder 36 based on the output of the probability model 37 present at decoder side, obtaining a quantized latent tensor zq. The quantized latent tensor is subjected to dequantization operation 38 to obtain a reconstructed latent tensor z. The reconstructed latent tensor is input to a decoder NN 39 to obtain a reconstructed input x, that is, a reconstructed version of the input x. The reconstructed input may also be referred to as reconstructed data, or reconstruction, or decoded data, or decoded input, or decoded output, or the like. Figure 2 is an example of an end-to-end learned codec, but a number of other designs or variations are possible.

[0094] The neural network components, or a subset of the neural network components, of an end-to-end learned codec may be trained by minimizing a rate-distortion loss function:L = D + AT?, where D is a distortion loss term, R is a rate loss term, and A is a weight that controls the balance between the two losses. The distortion loss term may be referred to also as reconstruction loss term, or simply reconstruction loss. The rate loss term may be referred to as rate loss. The distortion loss term measures the quality of the reconstructed or decoded output, and may include, but may not be limited to one or more of the following: mean square error (MSE), structure similarity (SSIM), MS-SSIM, losses derived from the use of a pretrained neural network, losses derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec or losses that are related to a performance of one or more machine analysis tasks or to an estimated performance of one or more machine analysis tasks.

[0095] As an example of losses derived from the use of a pretrained neural network, error(fl, f2) is a loss derived from the use of a pretrained neural network in which fl and f2 are the features extracted by a pretrained neural network for the input data and the decoded data, respectively, and er- ror() is an error or distance function, such as LI norm or L2 norm. An example of a loss derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec is an adversarial loss. An adversarial loss is the loss provided by a discriminator neural network that is trained adversarially with respect to the codec, following the settings proposed in the context of Generative Adversarial Networks (GANs) and their variants. As to a loss that is related to a performance of one or more machine analysis tasks or to an estimated performance of one or more machine analysis tasks, the one or more machine analysis tasks may include classification, object detection, image segmentation, instance segmentation, etc. In one example of this type of loss, the estimated performance of one or more machine analysis tasks may include a distortion computed based at least on a first set of features extracted from an output of the decoder and a second set of features extracted from a respective ground truth data. The first set of features and the second set of features in this example are output by one or more layers of a pretrained feature-extraction neural network.

[0096] Multiple distortion losses may be used and integrated into the distortion loss term D, such as a weighted sum of MSE and SSIM.

[0097] The rate loss term R may be used to train the encoder NN to output a low-entropy latent tensor, or a latent tensor such that the quantized latent tensor has low entropy, or a latent tensor such that the probability distribution of the quantized latent tensor can be better estimated or predicted by the probability model. The rate loss term R may be used to train the probability model to better estimate or predict the probability distribution of the quantized latent tensor. In one example, the rate loss term R is derived from the output of the probability model so as to represent the estimated entropy of the quantized latent representation, which indicates the number of bits necessary to represent the quantized latent tensor. The rate loss term R may also include a sparsification loss, that is, a loss that encourages the quantized latent tensor to comprise many zeros. Examples are L0 norm, LI norm, LI norm divided by L2 norm.

[0098] In order to train the neural network components, or a subset of the neural network components, of an end-to-end learned codec, one or more of reconstruction losses may be used, and one or more rate losses may be used. In one example, the one or more reconstruction losses and / or one or more rate losses are combined by means of a weighted sum. Typically, the different loss terms are weighted using different weights, and these weights determine how the final system performs in terms of rate-distortion performance. For example, if more weight is given to the reconstruction losses with respect to the rate losses, the system may learn to compress less but to reconstruct with higher accuracy (as measured by a metric that correlates with the reconstruction losses). These weights are usually considered to be hyper-parameters of the training process, and may be set manually by the persondesigning the training process, or automatically for example by grid search or by using additional neural networks.

[0099] In one case, the training process may be performed jointly with respect to the distortion loss D and the rate loss R. In another case, the training process may be performed in two alternating phases, where in a first phase only the distortion loss D may be used, and in a second phase only the rate loss R may be used.

[0100] For lossless video / image compression, the system may include only the probability model and the lossless encoder and lossless decoder. The loss function would include only the rate loss, since the distortion loss is always zero, such that no information is lost. As used herein, an inference phase, inference stage, inference time, or test time refer to the phase when a neural network or a codec is used for its purpose, such as encoding and decoding an input image.

[0101] Reducing the distortion in image and video compression is often intended to increase human perceptual quality, as humans are considered to be the end users, such as by consuming / watching the decoded images or videos. However, with the advent of machine learning, especially deep learning, there is a rising number of machines, e.g., autonomous agents, that analyze data independently from humans and that may even make decisions based on the analysis results without human intervention. Examples of such analysis are object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. For example, such analysis tasks may be performed by neural networks.

[0102] The device that performs the analysis may have multiple neural networks (NNs), also referenced as “machines”. These multiple machines may be used in combination in a manner determined, for example, by an orchestrator sub-system. The multiple machines may be used, for example, in succession, based on the output of the previously used machine, and / or in parallel. For example, a video may be analyzed by one machine, e.g., a first NN, for detecting pedestrians, by another machine, e.g., a second NN, for detecting cars, and by yet another machine, e.g., a third NN, for estimating the depth of all the pixels in the frames.

[0103] Example use cases and applications for such machines are self-driving cars, video surveillance cameras and public safety, smart sensor networks, smart television (TV) and smart advertisement, person re-identification, smart traffic monitoring, drones, etc. In addition to image and video data, automatic analysis and processing is increasingly performed for other types of data, such as audio, speech, text, etc.

[0104] Compressing (and decompressing) data where the end user comprises machines, e.g., neural networks, is commonly referred to as compression or coding for machines. In the case of video data, it is referred to as video compression or coding for machines (VCM). Compressing for machines may differ from compressing for humans, for example, with respect to the algorithms andtechnology used in the codec, or the training losses used to train any neural network components of the codec, or the evaluation methodology of codecs.

[0105] It is to be understood that, when considering the case of coding for machines, the terms “receiver- side” or “decoder-side” may refer to the physical or abstract entity or device which contains one or more machines, and runs these one or more machines on some encoded and eventually decoded video representation which is encoded by another physical or abstract entity or device, the “encoder-side device”.

[0106] Figure 3 is a general illustration of a pipeline of video coding for machines (VCM). A VCM encoder 42 encodes the input video into a bitstream. A bitrate may be computed from the bitstream, as a measure of the size of the bitstream. A VCM decoder 44 decodes the bitstream that was produced by the VCM encoder 42. The output of the VCM decoder 42 is referred in Figure 3 as “Decoded data for machines”. This data may be considered as the decoded or reconstructed video. However, in some implementations of this pipeline, this data may not have same or similar characteristics as the original video which was input to the VCM encoder 42. For example, this data may not be easily understandable by a human by simply rendering the data onto a screen. The output of VCM decoder 44 is then input to one or more task neural networks 46. In Figure 3, for the sake of illustrating that there may be any number of task-NNs 46, there are three example task-NNs, and a non-specified one (Task-NN X). One goal of VCM may be to obtain a low bitrate while guaranteeing that the task- NNs still perform well in terms of the evaluation metric associated to each task.

[0107] It is to be understood that, in some cases, the VCM decoder may not be present. In one example, the machines are run directly on the bitstream. In some other cases, the VCM decoder may comprise only a lossless decoding stage, and the lossless decoded data is provided as input to the machines. In yet some other cases, the VCM decoder may comprise a lossless decoding stage following by a dequantization operation, and the loss-decoded and dequantized data is provided as input to the machines.

[0108] When a conventional video encoder, such as a H.266 / VVC encoder, is used as a VCM encoder 42, any one of several approaches may be used to adapt the encoding to be suitable to machine analysis tasks. For example, a filter is used to preprocess the input to a conventional encoder. The filter may be a machine learning based filter, such as a convolutional neural network. In another example, one or more regions of interest (ROIs) may be detected using an ROI detection method. In this regard, ROI detection may be performed using a task NN, such as an object detection NN. In some cases, ROI boundaries of a group of pictures or an intra period may be spatially overlaid and rectangular areas may be formed to cover the ROI boundaries. The detected ROIs, such as rectangular areas, may be used in one or more way. For example, the quantization parameter (QP) may be adjusted spatially in a manner that ROIs are encoded using finer quantization step size(s) than other regions. For example, the QP may be adjusted CTU-wise. As other examples, the video maypreprocessed to contain only the ROIs, while the other areas are replaced by one or more constant values or removed, or the video may be preprocessed so that the areas outside the ROIs are blurred or filtered. In another example, a grid is formed in a manner that a single grid cell covers a ROI. Grid rows or grid columns that contain no ROIs are downsampled as preprocessing to encoding. In a further example, a quantization parameter of the highest temporal sublayer(s) is increased, e.g., coarser quantization is used, when compared to practices for human watchable video. In an additional example, the original video is temporally downsampled as preprocessing prior to encoding. A frame rate upsampling method may be used as postprocessing subsequent to decoding, if machine analysis at the original frame rate is desired.

[0109] In the context of video coding for machines, the terms “machine vision”, “machine vision task”, “machine task”, “machine analysis”, “machine analysis task”, “computer vision”, “computer vision task”, "task network" and “task” may be used interchangeably. Also, in the context of video coding for machines, the terms “machine consumption” and “machine analysis” may be used interchangeably.

[0110] A neural network may be used for filtering or processing input data. In this instance, the neural network may be referenced as a neural network based filter, an NN filter or a filter. An NN filter may include one or more neural networks, and optionally one or more components that may not be categorized as neural networks. An NN filter may perform various function including, but may not be limited to, visual enhancement, colourization, upsampling, super-resolution, inpainting, temporal extrapolation, generating content, or the like.

[0111] In some video codecs, a neural network may be used as filter in the encoding and decoding loop (also referred to simply as coding loop). In this instance, the neural network may be referred to as a neural network loop filter, or a neural network in-loop filter. The NN loop filter may replace all other loop filters of an existing video codec, or may represent an additional loop filter with respect to the already present loop filters in an existing video codec.

[0112] In one example, a codec is a modified VVC / H.266 compliant codec (e.g., a VVC / H.266 compliant codec that has been modified and, as a result, may no longer be compliant to the VVC / H.266 standard). In this example, the codec includes one or more NN loop filters. An input to the one or more NN loop filters may include at least a reconstructed block or frames (simply referred to as reconstruction) or data derived from a reconstructed block or frame, e.g., the output of a conventional loop filter. The reconstruction may be obtained based on predicting a block or frame, such as by intra-frame prediction or inter-frame prediction, and performing residual compensation. The one or more NN loop filters may enhance the quality of at least one of their input, so that a rate-distortion loss is decreased. The rate may indicate a bitrate (estimated or real) of the encoded video. The distortion may indicate a pixel fidelity distortion, such as mean-squared error (MSE), mean absolute error (MAE), mean average precision (mAP) computed based on the output of a task NN, such as an objectdetection NN, when the input is the output of the post-processing NN, or other machine task-related metric for tasks such as object tracking, video activity classification, video anomaly detection, etc. The enhancement may result into a coding gain, which can be expressed for example in terms of a Bjontegaard delta (BD) rate or a BD-peak signal to noise ratio (PSNR).

[0113] A neural network filter may be used as post-processing filter for a codec, such as by being applied to an output of an image or video decoder in order to remove or reduce coding artifacts. In one example, the NN filter is used as a post-processing filter in which the input includes data that is output by or is derived from an output of a traditional decoder, such as a decoder that is compliant with the VVC / H.266 standard. In another example, the NN filter is used as a post-processing filter in which the input includes data that is output by or is derived from an output of a decoder of an end-to- end learned decoder.

[0114] In the case of filtering images, an NN filter may take as input at least one or more first images to be filtered and may output at least one or more second images. The one or more second images are the filtered version of the one or more first images. In one example, the filter takes as input one image and outputs one image. In another example, the filter takes as input more than one image and outputs one image. In another example, the filter takes as input more than one image and outputs more than one image.

[0115] An NN filter may take as input also other data (also referred to as auxiliary data, or extra data) than the data that is to be filtered. The other data may be data that can aid the filter to perform a better filtering than if no auxiliary data was provided as input. In one example, the auxiliary data includes information about prediction data, and / or information about the picture type, and / or information about the slice type, and / or information about a quantization parameter (QP) used for encoding, and / or information about boundary strength, etc. In one example, the filter takes as input one image and other data associated with that image, such as information about the quantization parameter (QP) used for quantizing and / or dequantizing that image, and outputs one image.

[0116] An NN filter can be adapted at test time based at least on part of the data to be encoded and / or decoded and / or post-processed. Such adaption may also be referenced as content adaptation, overfitting, finetuning, optimization, specialization, and the like. The NN filter that results from the adaptation process may be referred to, for example, as an adapted filter, content-adapted filter, overfitted filter, finetuned filter, optimized filter, specialized filter, or the like. Although an NN filter is described herein for purposes of illustration, similar adaptation may be performed for other coding tools and / or post-processing tools that are based on neural network technology. For example, a neural network configured to perform intra-frame prediction, a neural network configured to perform interframe prediction, etc.

[0117] The overfitting process may be performed at the encoder side based on a training process. The resulting overfitted filter is then used to derive an overfitting signal, or adaptation signal. Theadaptation signal may be compressed and then signaled from the encoder to the decoder, in or along a bitstream that represents encoded data, such as an encoded image or video. The following figure illustrates an example of such encoder-side operations.

[0118] Figure 4 illustrates an example of the encoder-side operations. In Figure 4, x represents an input to the NN filter 50, x represents an output of the NN filter 50 and x represents a ground-truth data associated with x. The Compute loss function 52 computes a training loss I in order to overfit the NN filter 50, and the Overfit function 54 uses I to overfit the NN filter 50. As a result of the overfitting process, an overfitted NN filter is obtained, which is used, together with the original NN filter, to derive an adaptation signal at block 56. [Can you further describe how the filters as used as inputs to derive an overfitting signal?] The adaptation signal is compressed at block 58 and signaled at block 60 to a decoder or receiver.

[0119] At the decoder side, the overfitting signal, or a signal derived from the overfitting signal, is used to update the NN filter. The updated NN filter is then used to filter one or more pictures, or one or more blocks. This process is depicted in Figure 5 in which a compressed adaption signal is decompressed at block 62 in order to generate a decompressed adaption signal. At block 64, the original NN filter is then updated based on the decompressed adaption signal to generate an updated NN filter.

[0120] The NN filter that is obtained from the overfitting process at encoder side may be different from the NN filter that is obtained from the updating process at decoder side due to, for example, the adaptation signal being compressed in a lossy way. Thus, the former NN filter may be referred to as an overfitted filter, an adapted filter or the like, and the latter NN filter may be referred to as updated filter.

[0121] For the overfitting process performed on the encoder side, the adaptation process starts with an initial NN filter. The initial NN filter may be a pretrained NN filter, for example, which was pretrained during an offline stage on a sufficiently large dataset. In another example, the initial NN filter is a randomly initialized NN filter.

[0122] In the adaptation, one or more parameters of the NN filter may be adapted. Examples of the parameter(s) that may be adapted include, but are not limited to, the bias terms of a convolutional neural network, multiplier parameters that multiply one or more tensors produced by the NN filter, such as one or more feature tensors that are output by respective one or more layers of the NN filter, parameters of the kernels of a convolutional neural network parameters of an adapter layer and / or one or more arrays or tensors that are used as input to respective one or more layers of the NN filter.

[0123] The adaptation may be performed by means of a training process, e.g., by minimizing a loss function until a stopping criterion is met. The data used for this training process may include one or more pictures or blocks of input to the NN filter and associated respective one or more pictures or blocks of ground-truth data. In one example in which the filter is an in-loop filter, the input to the NN filter is reconstruction data, after prediction and residual compensation (e.g., after a decoded residualhas been added to or combined with a predicted picture or block) and the ground-truth data is the uncompressed data that is given as input to the encoder. In one example in which the filter is a post-pro- cessing filter, the input to the NN filter is decoded data (e.g., the output of a video decoder); and the ground-truth data is the uncompressed data that is given as input to the encoder.

[0124] The loss function used during the training process may include one or more distortion loss functions (also referred to as reconstruction loss functions) and zero or more rate loss functions. A rate loss function may measure, for example, the cost in terms of bitrate of signaling any adaptation signal, such as updates to the parameters of the NN filter. A distortion loss function may include one of MSE, multi-scale structural similarity index measure (MS-SSIM), video multimethod assessment fusion (VMAF), etc.

[0125] The adaptation signal may be derived based on the adapted NN filter and on the original NN filter, e.g., the NN filter before the overfitting process. In one example, the adaptation signal includes an update to one or more parameters of the NN filter. Such an update may also be referenced as a weight update, or parameter update. The update may be computed, for example, by subtracting the values of the adapted parameters, e.g., the parameters of the adapted NN filter, from the corresponding values of the original parameters, e.g., the parameters of the original NN filter. In another example, the adaptation signal includes the parameters (of the NN filter) that were adapted, also referred to as updated parameters, adapted parameters, adapted weights, or overfitted parameters, or the like.

[0126] In order to keep the size of the adaptation signal low, the adaptation signal may go through one or more compression steps, such as sparsification, quantization and lossless coding. In one example, an encoder that compresses the adaptation signal into a bitstream that is compliant with a neural network compression standard, such as MPEG NNC, may be used.

[0127] The compressed adaptation signal may be signaled from encoder to decoder in or along a bitstream that represents encoded image or video data. In one example, the compressed adaptation signal is signaled in an adaptation parameter set (APS) syntax structure of a video coding bitstream. In another example, the compressed adaptation signal is signaled in a supplemental enhancement lifor- mation (SEI) message of a video coding bitstream. Signaling may also include other information which is associated with the adaptation signal and that may be required for correctly parsing and / or decompressing and / or using the adaptation signal, such as any quantization parameters.

[0128] At the decoder side, the signaled compressed adaptation signal is received and decompressed. The decompressed adaptation signal may then be used to update the NN filter. In one example in which the adaptation signal has a weight update and the weight update includes one or more updates to respective one or more parameters of the NN filter, the one or more updates are added to the one or more parameters. In another example in which the adaptation signal includes one or more updated or adapted parameters, the one or more updated or adapted parameters are used to replacerespective one or more parameters of the NN filter. Once the NN filter has been updated based on the adaptation signal, the updated NN filter may be used for its purpose. For example, for filtering an input picture or an input block.

[0129] An example architecture of a neural network based in-loop filter that is included in a video decoder is depicted in Figure 6. In this example, “luma” and “chroma” refer to the reconstructed luma and chroma, respectively, that are to be enhanced by the NN filter, and may represent an intermediate result of an encoding or decoding operation. In one example in which the filter is a loop filter of a video codec, “luma” and “chroma” may represent the result of combining a predicted block with a decoded residual. The luma and chroma, that may be extracted from a YUV420 signal, are provided as input to a Discrete Cosine Transform (DCT), obtaining DCT-transformed luma and DCT-trans- formed chroma that are then concatenated to form the reconstruction Rec.

[0130] The terms Rec, Pred, BS, BaseQP, SliceQP and IPB represent the inputs to the NN filter and each of those inputs is usually in the format of a tensor of shape BxCxHxW in which B indicates a batch size, e.g., number of pictures or blocks, C indicates a number of channels, and H and W indicate a height and width, respectively. The square brackets and the number within them, e.g., Rec[3], indicate the number of channels of the associated tensor. For example, Rec[3] indicates that the input tensor Rec contains 3 channels and thus has shape Bx3xHxW, where the three channels may represent the luma channel, the Blue- Yellow Chrominance (Cb) channel and the Red-Green Chrominance (Cr) channel. The Cb channel and the Cr channel may be collectively referred to as chroma.

[0131] Pred refers to prediction. BS refers to boundary strength, BaseQP refers to the sequencelevel quantization parameter (QP), SliceQP refers to the slice-level QP and IPB refers to the type of slice or type of picture, e.g., intra slice, P inter slice or B inter slice). In some embodiments, Rec may be referred to as main input, or data to be filtered, whereas Pred, BS, BaseQP, SliceQP and IPB may be referred to as auxiliary input, or data not to be filtered.

[0132] Each block in Figure 6 represents an operation, such as one or more NN layers. By way of example, the block “ConvKlxK2,Z” indicates a convolutional layer with kernel size KlxK2 and a number of kernels equal to Z. When present, the term “+ PReLU” indicates that a layer is followed by a Parameterized Rectified Linear Unit (PReLU). When present, the term “PReLU +” indicates that a layer is preceded by a Parameterized Rectified Linear Unit (PReLU). When present, the term “s=2” indicates that a convolutional layer has stride equal to 2; when not present, the convolutional layer has stride equal to 1. “Split” refers to an operation that splits a tensor across the channel dimension.

[0133] “Luma BB” 70 and “Chroma BB” 72 indicate a backbone block used for filtering or processing the luma channel and the chroma channel, respectively. The architecture of a backbone block 74 is also shown in Figure 6 and includes several layers and operations. “SepConv3x3” indicates a block 76 that comprises a separable convolution. An illustration of the SepConv3x3 block 76 is also shown in Figure 6 and includes several layers. “PS” refers to a Pixel Shuffle operation, whichrearranges elements in a tensor of shape Bx(C*r*r)xHxW to a tensor of shape BxCx(H*r)x(W*r), where r is an upscale factor. “LumaOut” and “ChromaOut” represent the filtered luma and the filtered chroma, respectively, e.g., the final outputs from the NN filter.

[0134] By way of illustration, the NN architecture is figuratively organized into the following sections: head, fuse, transition, luma backbone, chroma backbone, luma tail 77 and chroma tail 78. However, it is to be noted that other possible organizations of the NN into subsets or blocks or sections may be possible and may still be in the scope of the present disclosure.

[0135] The input tensors are input to respective convolutional layers that are part of the “head” section of the NN filter. The outputs of those convolutional layers are tensors, referred to head tensors. As part of the operations of the “fuse” section of the NN, the head tensors are concatenated into a single tensor across the channel dimension. It is noted, however, that the concatenation operation is not illustrated. The concatenated tensor is input to a convolutional layer, followed by a non-linear activation function PReLU. The output of the fuse section is input to the “transition” section of the NN, which includes a separable convolutional layer with stride equal to 2 and a convolutional layer and a PReLU layer. The output of the transition section is a tensor of shape Bx(2*C)x(H / 2)x(W / 2), and the tensor is split into two sub-tensors, where each subtensor is of shape BxCx(H / 2)x(W / 2). A first subtensor is used to filter the luma and a second subtensor is used to filter the chroma. The first subtensor is input to a convolutional layer that maps the number of channel C to a different number of channels, followed the “luma backbone” section 70, and the second subtensor is input to a convolutional layer that maps the number of channel C to a different number of channels, followed by the “chroma backbone” section 72. The luma backbone section 70 includes Ny luma backbone blocks, and the chroma backbone section 72 includes Nuv chroma backbone blocks. The output of a backbone block is input to the next backbone block, until the last backbone block in the section. The output of the last luma backbone block is input to the “luma tail” section 77, that includes a 1x1 convolutional layer, a separable convolution SepConv3x3, another 1x1 convolutional layer and a PReLU operation, a Conv3x3 layer, a Pixel Shuffle (PS) operation, and an inverse DCT operation. The output of the PixelShuffle operation is added to the input luma, in order to obtain LumaOut. The output of the last chroma backbone block is input to the “chroma tail” section 78, that includes a 1x1 convolutional layer, a separable convolution SepConv3x3, another 1x1 convolutional layer and a PReLU operation, a Conv3x3 layer and an inverse DCT operation. The output of the inverse DCT operation is added to the input chroma, in order to obtain ChromaOut.

[0136] Figure 6 also provides examples of the values of various hyper-parameters of the NN, such as the number of channels of convolutional layers designated DI, D2, D3, D4, D5, D6, C and Cl, and the number of luma and chroma backbone blocks Ny and Nuv.

[0137] As used herein, frame, picture and image are used interchangeably. The input and output to an end-to-end learned codec may be any type of content, such as audio or video, e.g., pictures. Assuch, the input and output of a NN filter may be pictures. In some embodiments, when the term block refers to a portion of a picture, the block may alternatively be referred to as a frame, a picture or image. In other words, at least some embodiments, even when described as applied to a picture, may be applicable also to a block, e.g., to a portion of a picture.

[0138] As noted above, neural networks are being increasingly used as part of encoding and / or decoding pipelines, such as in image of video codecs. One example is a neural network based in-loop filter. Other examples include a NN post-processing filter or the NN may be part of an intra-frame prediction process, an inter-frame prediction process, a transformation process, such as a transform of a prediction residual signal, and / or an end-to-end learned coded, such as a NN that receives a lossless- decoded latent tensor and outputs reconstructed or decoded data. The NN may also perform spatial upsampling and / or super -resolution. A method, apparatus and computer program product are provided to improve the performance of such neural networks, in terms of one or more performance metrics, in order to correspondingly improve the overall performance of the encoder, such as a video encoder, and / or decoder, such as a video decoder, that includes the neural network.

[0139] By way of example, but not of limitation, the apparatus, method and computer program product of an example embodiment are described as applied to a NN filter that enhances or improves the quality of one of its inputs. One example is a NN-based in-loop filter that enhances the quality of an input picture or block or patch in terms of one or more quality metrics, such as the Peak Signal-to- Noise Ratio (PSNR) metric. However, the apparatus, method and computer program product may be applied to other NNs than a NN filter.

[0140] Also, the apparatus, method and computer program product of an example embodiment may be applied to a NN that is used for decoding a data item. However, other embodiments of the apparatus, method and computer program product may include a NN that is used for post-processing a decoded data item or data derived from a decoded data item.

[0141] While at least some embodiments are described in relation to input and output data in the form of images or video frames or pictures, the apparatus, method and computer program product of other embodiments may be applicable also to other types of data, such as audio frames. Furthermore, while at least some embodiments are described by considering a full image, other embodiments of the apparatus, method and computer program product may be applicable also to one or more blocks or portions of an image.

[0142] It is to be noted that in some cases an encoder includes a decoder or a subset of components of a decoder. For example, a NN-based in-loop filter may be present in both the encoder and the decoder of a codec. In at least some embodiments, a processing unit or processing block may be part of a NN that may be used within an encoder and / or a decoder such as a NN-based in-loop filter or a NN-based post-processing filter. For example, an individual processing unit of an embodiment may be used within an example architecture of an NN in-loop filter.

[0143] As described above, the apparatus of an example embodiment may be embodied by an encoder or decoder, but the apparatus may alternatively be embodied by any of a variety of other computing devices. Regardless of the device that embodies the apparatus, the apparatus 110 of an example embodiment depicted in Figure 7 may include, for example, at least one processor 112 and at least one memory 114 storing instructions 115 that, when executed by the at least one processor, cause the apparatus 110 at least to perform the method(s) as disclosed herein. In an example, the at least one memory 114 and the instructions 115 (e.g. a computer program code, software), are configured, with the at least one processor 112, to cause the apparatus 110 to perform the method(s) as disclosed herein, and any of the embodiments thereof. For example, the apparatus 110 may be caused or configured to perform at least the method of Figure 8 and / or any one or more of the embodiments described.

[0144] A processor 112 may comprise circuitry, or be constituted as circuitry or circuitries, the circuitry or circuitries being configured to perform phases of methods in accordance with example embodiments described herein. As used in this application, the term “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations, such as implementations in only analog and / or digital circuitry, and (b) combinations of hardware circuits and software, such as, as applicable: (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a user equipment, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation. This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0145] The memory 114 may be implemented using any suitable data storage technology. The memory may comprise a database for storing data. The memory 114 may be at least in part external to apparatus 110 but accessible to apparatus 110. The instructions 115 may be comprised in a computer readable medium or a non-transitory computer readable medium. A term non-transitory, as used herein, is a limitation of the medium itself (e.g., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., random access memory (RAM) vs. read only memory (ROM)).

[0146] The apparatus 110 of an example embodiment may include a radio interface 116. The radio interface 116 may provide the apparatus 110 with communication capabilities. The radio interface116 may comprise a receiver configured to receive information in accordance with at least one cellular or non-cellular standard. The radio interface 116 may comprise a transmitter configured to transmit information in accordance with at least one cellular or non-cellular standard. The receiver may comprise more than one receiver. The transmitter may comprise more than one transmitter. The radio interface 116 may comprise a transceiver configured to receive and transmit information in accordance with at least one cellular or non-cellular standard. The transceiver may comprise more than one transceiver.

[0147] The apparatus 110 may optionally comprise a user interface 118 comprising, for example, at least one of a keypad, a microphone, a touch display, a display, a speaker, etc. The user interface 118 may be used to control the apparatus by the user. The user interface 118 may be external to the apparatus 110. For example, the apparatus 110 may be connected to another device, such as a computer, either via wireless or wired connection, and the apparatus 110 is controlled by the user via the computer.

[0148] The operations performed by the apparatus 110 in accordance with an example embodiment are shown in Figure 8. In this regard, the apparatus 110 includes means, such as the at least one processor 112 or the like, for generating the two or more copies of the input and / or dividing an input into two or more portions and means, such as the at least one processor 112 or the like, for differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input to generate respective intermediate outputs. See blocks 120 and 122 of Figure 8. The apparatus 110 of this example embodiment also includes means, such as the at least one processor 112 or the like, for combining the respective intermediate outputs to obtain an output or data from which the output is derivable. See block 124 of Figure 8.

[0149] In this example embodiment, the instructions, when executed by the at least one processor 112, generate a distributed processing unit that includes a demultiplexer, such as a splitter, a selector, a distributor, a router or a switch, configured to divide the input into two or more portions and / or generate two or more copies of the input, two or more processing branches configured to differently process the respective copies of the two or more copies and / or portions of the two or more portions of the input and a combiner configured to combine the respective intermediate outputs. The processing unit or processing block may include at least one distributed processing unit. Additionally, the instructions, when executed by the at least one processor 112, define at least one or more neural network layers that are configured to serve at least as the two or more processing branches.

[0150] The input data item may be represented as a tensor. In one example, the input data item may comprise a 3-dimensional tensor, where two dimensions may represent spatial dimensions, e.g., vertical and horizontal dimensions, and another dimension may represent or be referred to as a channel dimension. In another example, the input data item may comprise a 4-dimensional tensor, where two dimensions may represent spatial dimensions, e.g., vertical and horizontal dimensions, anotherdimension may represent or be referred to as a channel dimension, and yet another dimension may represent or be referred to as temporal dimension.

[0151] Differences among the two or more portions of the input data item may include, but may not be limited to, one or more of the following: channels, spatial location and / or temporal location. In other words, the demultiplexer may divide or decompose the input data item into two or more portions along one or more dimensions of the tensor or data structure representing the input data item. In one example, the two or more portions of the input data item may include respective two or more subsets of channels, where any two of the two or more subsets of channels may be either disjoint or at least partially overlapping. In another example, the two or more portions of the input data item may include respective two or more spatial regions, where any two of the two or more spatial regions may be either disjoint or at least partially overlapping.

[0152] Although the demultiplexer is described herein to decompose an input tensor based on the channel dimension, e.g., the demultiplexer outputs two tensors that comprise respective two subsets of channels of the input tensor to the demultiplexer, the demultiplexer may decompose the input tensor in other manners, such as by decomposing the input tensor, such as spatially (e.g., in the case of an input tensor representing a visual scene, dividing the input tensor into two tensors representing respective two spatial regions of the visual scene). In one example, the demultiplexer may perform a tensor slicing operation. In some embodiments, the demultiplexer may be configured to output two or more signals that are copies of, the same as or substantially the same as an input to the demultiplexer, e.g., the two or more portions of the input data item may include respective two or more copies of the input data item. The combiner may be configured to combine the respective intermediate outputs in any of a variety of manners including, for example, concatenation, summation and / or stacking.

[0153] By way of example, Figure 9 illustrates an example of a processing unit 130 implemented by the apparatus 110 of Figure 7 in accordance with one embodiment. In this embodiment, the processing unit 130 includes a distributed processing unit 132 with the input data item (“input”) to the processing unit 130 serving as an input to the distributed processing unit 132 and an output of the distributed processing unit 132 serving as an output of the processing unit 130 (“output”). The demultiplexer 134 divides or distributes the input to the distributed processing unit 132 into two portions, where the two portions are processed by respective two processing branches 136 designated “Processing branch 1” and “Processing branch 2”. The intermediate outputs from the two processing branches 136 are combined by the combiner 138 to obtain the output of the distributed processing unit 132.

[0154] By way of a further example, Figure 10 illustrates another embodiment of a processing unit 140 implemented by the apparatus 110 of Figure 7. In this embodiment, the processing unit 140 includes one distributed processing unit 142. The input data item (“input”) to the processing unit 140 is processed by the at least one processor 112 to derive input to the distributed processing unit 142 asshown in block 144 and the output of the distributed processing unit 142 is processed by the at least one processor 112 to derive an output (“output”) of the processing unit as shown in block 146. The demultiplexer 148 divides or distributes the input to the distributed processing unit 142 into two portions, where the two portions are processed by respective two processing branches 150, 152 designated as “Processing branch 1” and “Processing branch 2”. Intermediate outputs from the two processing branches are combined by a combiner 154 to obtain the output of the distributed processing unit 142.

[0155] In one embodiment, the input to the distributed processing unit 142 may be derived or provided in block 144 by a second distributed processing unit. The output of the distributed processing unit 142 of this example embodiment may additionally or alternatively be processed by a third distributed processing unit in order to derive the output of the processing unit 140 as shown in block 146.

[0156] In an example embodiment, a processing branch, e.g., at least one of the two or more processing branches, may be formed of one or more neural network layers. In one example, a processing branch may include a convolutional layer. In another example, a processing branch may be configured to perform an identity function such that no computations on the input are performed by the respective processing branch, thus copying its input to its output. In some cases, the identity function may be referred to as a skip connection or a residual connection.

[0157] In one example, the two or more portions of the input to the demultiplexer 148 include two disjoint subsets of channels. In this embodiment, the combiner may be configured to perform a concatenation operation along the channel dimension. Figure 11 illustrates this example, where the demultiplexer is represented by a “Split channels” block 160 and the combiner performs a concatenation operation as represented by a “Concaf ’ block 162.

[0158] When the input to the distributed processing unit 164 is a 3-dimensional tensor with C channels, height equal to H and width equal to W, and the demultiplexer represented by the “Split channels block 160 divides the input into two disjoint subsets of the input, the “Split channels” block 160 outputs a first 3-dimensional tensor with Ca channels, height equal to H and width equal to W, and a second 3 -dimensional tensor with Cb channels, height equal to H and width equal to W, where the sum of Ca and Cb is equal to C. The first 3-dimensional tensor is input to a first processing branch 166, and the second 3-dimensional tensor is input to a second processing branch 168.

[0159] The first processing branch 166 includes a first convolutional layer, represented by a “ConvKlxK2, Ca” block, where the naming indicates that the first convolutional layer comprises a number of kernels equal to Ca and where each of the Ca kernels has spatial size (or simply kernel size) equal to KI by K2. The second processing branch 168 includes a second convolutional layer, represented by a “ConvK3xK4, Cb” block, where the naming indicates that the second convolutional layer comprises a number of kernels equal to Cb and where each of the Cb kernels has spatial size (orsimply kernel size) equal to K3 by K4. This same (or similar) naming convention for convolutional layers will be used for other embodiments or examples in the present disclosure. However, in some other examples and figures, the kernel size of a convolutional layer may not be specified, for example as “Conv, Ca”. While in this example illustrated in Figure 11, the number of channels of the input to a processing branch is equal to the number of channels of the (intermediate) output of the processing branch, and the number of channels of the input to the distributed processing unit is equal to the number of channels of the output of the distributed processing unit, this may not be the case in general. For example, the number of channels of the input to a processing branch may be different than the number of channels of the (intermediate) output of the processing branch.

[0160] In one embodiment, at least one of the two or more processing branches comprises a modulation operation based on one or more multiplier parameters, where an input to the at least one of the two or more processing branches is multiplied (or scaled) by the one or more multiplier parameters.

[0161] In one example, an input to the at least one of the two or more processing branches is a 3- dimensional tensor with C channels, height equal to H and width equal to W; a demultiplexer splits the input tensor into a first tensor with Ca channels and a second tensor with Cb channels. The first tensor is input to a first processing branch, and the second tensor is input to a second processing branch, where the second processing branch includes multiplying the Cb channels of the second tensor by respective Cb multiplier parameters.

[0162] Figure 12 illustrates an example of this embodiment, where the first processing branch 170 includes a convolutional layer with Ca kernels (“Conv, Ca”) and the second processing branch 172 includes a modulation operation based on multiplier parameters (“Multipliers”).

[0163] Figure 13 illustrates a further example of this embodiment, where a distributed processing unit 180 includes three processing branches, thus the demultiplexer (denoted as “Split channels”) 182 divides its input tensor into a first tensor with Ca channels, a second tensor with Cb channels and a third tensor with Cc channels; the first tensor is input to first processing branch 184 that comprises a convolutional layer (similar to the example of Figure 12); the second tensor is input to a second processing branch 186 that comprises a modulation operation based on multiplier parameters (similar to the example of Figure 13); the third processing branch 188 is a skip connection (also known as an identity function or a copy function) such that the third tensor is not modified and, is instead, concatenated with the outputs of the first processing branch 184 and the second processing branch 186 by the Concat block 190.

[0164] Figure 14 illustrates a further example of this embodiment, where one of the previously described examples of a distributed processing unit is embodied into a processing unit, where the processing unit represents a backbone block that is part of a neural network based in-loop filter. This processing unit 200 comprises one distributed processing unit 202. An input to the distributedprocessing unit 202 is derived from the input to the processing unit (“Input”) 200 by means of a PReLU layer and a “Convlxl,C” layer (e.g., a convolutional layer with kernel size 1x1 and C kernels) 204. An output of the distributed processing unit 202 is input to a “Convlxl,Cl” layer (e.g., a convolutional layer with kernel size 1x1 and Cl kernels) 206, and the output of this layer is added to the input to the processing unit 200 to obtain the output of the processing unit.

[0165] In some embodiments, a processing branch may apply additive parameters. For example, at least one of the two or more processing branches may include a modulation operation based on one or more additive parameters, where an input to the at least one of the two or more processing branches is added to the one or more additive parameters. In one embodiment, modulations based on both multiplier parameters and additive parameters may be performed within a distributed processing unit.

[0166] As shown in the example of Figure 15, additive parameters may be used in one processing branch and multiplier parameters may be used in another processing branch. As shown, a distributed processing unit 210 comprises three processing branches. An input tensor is split into three tensors as indicated by the Split channels block 211 with a first tensor comprising Ca channels being input to a first processing branch 212 that comprises a convolutional layer with Ca kernels; a second tensor comprising Cb channels being input to a second processing branch 214 that comprises multiplier parameters; a third tensor comprising Cc channels being input to a third processing branch 216 that comprises additive parameters. Outputs of the three processing branches are concatenated by block Concat 218 to obtain an output of the distributed processing unit 210.

[0167] In another example depicted in Figure 16, multiplier parameters and additive parameters may be used within the same processing branch 220. In this example, a distributed processing unit 222 comprises two processing branches. An input tensor is split into two tensors as indicated by the Split channels block 224: a first tensor comprises Ca channels and is input to a first processing branch that comprises a convolutional layer 226 with Ca kernels; and a second tensor comprises Cb channels and is input to a second processing branch 220 that comprises multiplier parameters and additive parameters. The multiplier parameters and the additive parameters may be used in a sequential manner, e.g., the second tensor is multiplied by the multiplier parameters and the resulting tensor is added to the additive multipliers. Outputs of the two processing branches are concatenated by the Concat block 228 to obtain an output of the distributed processing unit 222.

[0168] In one embodiment, at least one of the two or more processing branches comprises a convolutional layer with a kernel size equal to 1x1 (1 by 1). Figure 17 illustrates an example of this embodiment in which a distributed processing unit 230 comprises two processing branches. A first processing branch 232 comprises a first convolutional layer “Conv, Ca” with Ca kernels. A second processing branch 234 comprises a second convolutional layer “Convlxl, Cb” with Cb kernels and kernel size equal to 1x1 (1 by 1). The distributed processing unit 230 also includes a demultiplexer tosplit the channels in block 236 and a combiner to concatenate the intermediate outputs as in block 238.

[0169] Figure 18 illustrates a further example of this embodiment, where the distributed processing unit 240 is used within a processing unit 242 comprised in an NN-based in-loop filter, such as a backbone block. In this embodiment, a first processing branch 243 of the distributed processing unit 240 comprises a first convolutional layer “Conv3x3,Ca” with Ca kernels and kernel size equal to 3x3, and a second processing branch 244 of the distributed processing unit comprises a second convolutional layer “Convlxl,Cb” with Cb kernels and kernel size equal to 1x1. The processing unit 242 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0170] In one embodiment, at least one of the two or more processing branches comprises a grouped convolutional layer, e.g., a convolutional layer where different kernels are applied to different groups or subsets of channels of an input tensor. Figure 19 illustrates an example of this embodiment, where “GroupConv, Cb” block 250 indicates a grouped convolutional layer with Cb kernals. The processing unit 252 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0171] Figure 20 illustrates a further example of this embodiment, where the distributed processing unit 260 is used within a processing unit 262 representing a backbone block comprised in an NN-based in-loop filter. The processing unit 252 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0172] In one further example, a grouped convolutional layer comprised in a processing branch of a distributed processing unit may comprise a group size equal to 1. Thus, the channels in the input tensor to this layer are grouped into a number of groups equal to the number of channels of the input tensor. This may be referred to also as a depth- wise convolutional layer. Figure 21 illustrates this example, where “DepthWiseConv, Cb” denotes a depth-wise convolutional layer 270 with Cb kernels and Cb groups.

[0173] Some embodiments may also provide for multiscale processing within a distributed processing unit. In one embodiment, a distributed processing unit may comprise two or more processing branches where at least two of the two or more processing branches comprise respective at least two scales or resolutions, e.g., the at least two of the two or more processing branches process data at respective at least two scales or resolutions. A distributed processing unit of this embodiment may be referenced as a multiscale distributed processing unit in some examples.

[0174] In one example, a processing branch may comprise decreasing a scale or resolution of its input to obtain data at a lower scale or resolution, processing the data at a lower scale or resolution to obtain processed data, increasing a scale or resolution of the processed data to obtain processed data at a higher scale or resolution, e.g., at same scale or resolution as an input to the processing branch, sothat the input to the processing branch comprises the same scale or resolution as the output of the processing branch.

[0175] A scale or resolution may be decreased by means of a downsampling operation, or by means of a convolutional layer with stride greater than 1, or by means of a pixel unshuffle operation, or by means of an interpolation-based operation, or by other suitable means. A scale or resolution may be increased by means of an upsampling operation, or by means of a transpose convolutional layer with stride greater than 1, or by means of a pixel shuffle operation, or by means of an interpolationbased operation, or by other suitable means.

[0176] In another example, a processing branch may comprise processing its input or data derived from its input by using at least one convolutional layer with stride greater than 1. In yet another example, a processing branch may comprise processing its input or data derived from its input by using at least one transpose convolutional layer with stride greater than 1. In yet another example, a processing branch may comprise processing its input or data derived from its input by using at least one convolutional layer with stride greater than 1 and one transpose convolutional layer with stride greater than 1.

[0177] In one example, a first processing branch comprises a first scale or resolution that is the same as a scale or resolution of an input to the distributed processing unit, and a second processing branch comprises a second scale or resolution that is lower than the scale or resolution of an input to the distributed processing unit. Figure 22 illustrates an embodiment of this example. In this embodiment, the “Downsample” block 280 performs a downsampling operation with a downsampling factor equal to 2; the “Upsample” block 282 performs an upsampling operation with an upsampling factor equal to 2; the “Conv, Ca” block 284 refers to a convolutional layer with Ca kernels applied to data with the same scale or resolution as an input to the distributed processing unit; the “Conv, Cb” block 286 refers to a convolutional layer with Cb kernels applied to data with lower scale or resolution than the scale or resolution of the input to the distributed processing unit.

[0178] Figure 23 illustrates a further example of this embodiment, where the distributed processing unit 290 is used within a processing unit 290 and represents a backbone block comprised in an NN-based in-loop filter. The processing unit 292 includes other components as depicted in the prior figures and described in relation to the prior embodiments. In this embodiment, the “SepConv3x3, Ca” block 294 denotes a separable convolutional layer or a separable and depth-wise convolutional layer, where the kernel size is equal to 3x3 and the number of kernels is equal to Ca. Similarly, the “SepConv3x3, Cb” 296 denotes a separable convolutional layer or a separable and depth-wise convolutional layer, where the kernel size is equal to 3x3 and the number of kernels is equal to Cb. In one example, a separable convolutional layer denoted as “SepConvKlxK2” comprises a first convolutional layer with a horizontal kernel of size lxK2 followed by a second convolutional layer with a vertical kernel of size Klxl. In another example, a separable convolutional layer comprises a firstconvolutional layer with a vertical kernel of size Klxl followed by a second convolutional layer with a horizontal kernel of size lxK2.

[0179] In one embodiment, a multiscale distributed processing unit may comprise a processing branch that comprises a modulation operation based on one or more multiplier parameters. Figure 24 illustrates an example of this embodiment, where a first processing branch comprises a first scale or resolution that is the same as a scale or resolution of an input to the distributed processing unit 300, and a second processing branch comprises a second scale or resolution that is lower than the scale or resolution of an input to the distributed processing unit. The first processing branch 302 comprises a convolutional layer with Ca kernels. The second processing branch 304 comprises a modulation operation based on Cb multiplier parameters (denoted as “Multipliers”), which multiplies the input tensor by the Cb multiplier parameters.

[0180] Figure 25 illustrates a further example of this embodiment, where the distributed processing unit 310 is used within a processing unit 312 representing a backbone block comprised in an NN-based in-loop filter. The processing unit 312 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0181] In one embodiment, a multiscale distributed processing unit may comprise a processing branch that comprises a skip connection, an identity function or a copy operation. Figure 26 illustrates an example of this embodiment in which the processing unit 320 comprises a (multiscale) distributed processing unit. A demultiplexer (“Split channels” block) 322 splits an input tensor (that is output by a convolutional layer “Conv, C”) into a first tensor and a second tensor along the channel dimension, where the input tensor comprises C channels, the first tensor comprises Cb channels and the second tensor comprises Ca channels. The (multiscale) distributed processing unit 322 comprises a first processing branch and a second processing branch. The first processing branch comprises a first scale or resolution that is same as a scale or resolution of an input to the distributed processing unit. The second processing branch comprises a second scale or resolution that is lower than the scale or resolution of an input to the distributed processing unit. The first processing branch 324 comprises a skip connection, an identity function or a copy operation, applied to the first tensor with Cb channels. The second processing branch 326 comprises a convolutional layer (“Conv, Ca” block) with Ca kernels applied to the second tensor with Ca channels.

[0182] Figure 27 illustrates a further example of this embodiment, where the previously described processing unit 320 is used within a more complex processing unit representing a backbone block comprised in an NN-based in-loop filter. The processing unit 320 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0183] Some embodiments include processing branches with different directional processing. In one embodiment, a processing unit may comprise two or more processing branches where at least two of the two or more processing branches comprise processing data with respective at least twodirections, e.g., the at least two of the two or more processing branches process their respective at least two inputs based on respective at least two different directions.

[0184] Figure 28 illustrates an example of this embodiment. In this example, a distributed processing unit 330 comprises a first processing branch and a second processing branch, where the first processing branch 332 comprises a horizontal convolutional layer (e.g., a convolutional layer with kernel size equal to IxKl) with Ca kernels and the second processing branch 334 comprises a vertical convolutional layer (e.g., a convolutional layer with a kernel size equal to K2xl) with Cb kernels.

[0185] Figure 29 illustrates a further example of this embodiment, where the distributed processing unit 340 is used within a processing unit 32 representing a backbone block comprised in an NN-based in-loop filter. The processing unit 342 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0186] In other embodiments, the processing branches may provide for both different directional and multiscale processing. In one embodiment, a distributed processing unit may comprise two or more processing branches where at least two of the two or more processing branches comprise processing data with respective at least two directions (e.g., the at least two of the two or more processing branches process their respective at least two inputs based on respective at least two different directions), and where one or more of the at least two of the two or more processing branches comprise two or more processing sub-branches where at least two of the two or more processing sub-branches comprise respective at least two scales or resolutions, e.g., the at least two of the two or more processing sub-branches process data at respective at least two scales or resolutions. In order words, one or more of the at least two of the two or more processing branches comprise a multiscale processing.

[0187] Figure 30 illustrates an example of this embodiment that includes distributed processing unit 350 comprising a first processing branch and a second processing branch. An input to the first processing branch comprises a first tensor with Ca channels. An input to the second processing branch comprises a second tensor with Cb channels. The first processing branch performs vertical processing, by using convolutional layers with vertical kernels. The second processing branch performs horizontal processing, by using convolutional layers with horizontal kernels. The first processing branch comprises a first processing sub-branch and a second processing sub-branch, where the first processing sub-branch processes data at a same scale or resolution as a scale or resolution of an input to the distributed processing unit and the second processing sub-branch processes data at a different scale than a scale of an input to the distributed processing data. The second processing branch comprises a third processing sub-branch and a fourth processing sub-branch, where the third processing sub-branch processes data at a same scale or resolution as a scale or resolution of an input to the distributed processing unit and the fourth processing sub-branch processes data at a different scale than a scale of an input to the distributed processing data. The first processing sub-branch 352 comprises a first convolutional layer with Ca kernels and kernel size equal to IxKl. The second processing sub-branch 354comprises a second convolutional layer with Ca kernels and kernel size equal to lxK2. The third processing sub-branch 356 comprises a third convolutional layer with Cb kernels and kernel size equal to K3xl. The fourth processing sub-branch 358 comprises a fourth convolutional layer with Ca kernels and kernel size equal to K4xl. The distributed processing unit 350 includes other components (e.g., downsample, upsample, summation operation, concatenation operation, split channels operation, etc.) as depicted in the prior figures and described in relation to the prior embodiments.

[0188] Please note that KI, K2, K3, K4 may be equal to any suitable value, such as any integer value. Thus, any two or more of the values of KI, K2, K3, K4 may be equal or may be different.

[0189] Figure 31 illustrates a further example of this embodiment, where the distributed processing unit 360 is used within a processing unit 362 comprised in an NN-based in-loop filter, for example, within a backbone block (illustrated and described generally in relation to Figure 6) for the architecture of NN-based in-loop filter. The distributed processing unit 360 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0190] In one embodiment, a distributed processing unit may comprise two or more processing branches where at least one of the two or more processing branches comprises two or more processing sub-branches with respective two or more inputs to the two or more processing sub-branches that are derived from an input to the at least one of the two or more processing branches, and where the two or more processing sub-branches comprise processing the respective two or more inputs in respective two or more directions to obtain two or more outputs of the respective two or more processing subbranches, and where the two or more outputs of the respective two or more processing sub-branches are combined to obtain an output of the at least one of the two or more processing branches based on a combination operation.

[0191] In one example, a processing branch comprises a first processing sub-branch and a second processing sub-branch. The first processing sub-branch comprises a first convolutional layer with horizontal kernel and the second processing sub-branch comprises a second convolutional layer with vertical kernel.

[0192] In one embodiment, the two or more inputs to the respective two or more processing subbranches comprised in a particular processing branch are the same or substantially the same as an input to the particular processing branch. In another embodiment, the two or more inputs to the respective two or more processing sub-branches comprised in a particular processing branch are different from each other and different from an input to the particular processing branch. For example, the two or more inputs may comprise respective two or more subsets of channels of the tensor that is input to the particular processing branch.

[0193] In one embodiment, a distributed processing unit may comprise two or more processing branches where at least two of the two or more processing branches comprise respective two scales or resolutions, e.g., the at least two of the two or more processing branches process data at respectivetwo scales or resolutions, and where at least one of the two or more processing branches comprises two or more processing sub-branches with respective two or more inputs to the two or more processing sub-branches that are derived from an input to the at least one of the two or more processing branches, and where the two or more processing sub-branches comprise processing the respective two or more inputs in respective two or more directions to obtain two or more outputs of the respective two or more processing sub-branches, and where the two or more outputs of the respective two or more processing sub-branches are combined to obtain an output of the at least one of the two or more processing branches based on a combination operation.

[0194] Figure 32 illustrates an example of this embodiment in which a distributed processing unit 370 comprises two processing branches comprising respective two scales or resolutions. A first processing branch comprises a first processing sub-branch, a second processing sub-branch and a first summation operation. A second processing branch comprises a downsample operation, a third processing sub-branch, a fourth processing sub-branch, a second summation operation and an upsample operation. The first processing branch, the first processing sub-branch and the second processing subbranch are applied on an input tensor comprising Ca channels. The second processing branch, the third processing sub-branch and the fourth processing sub-branch are applied on an input tensor comprising Cb channels. The first processing sub-branch 372 comprises a first horizontal convolutional layer with a kernel size equal to 1x3. The second processing sub-branch 374 comprises a first vertical convolutional layer with a kernel size equal to 3x1. The third processing sub-branch 376 comprises a second horizontal convolutional layer with a kernel size equal to 1x3. The fourth processing subbranch 378 comprises a second vertical convolutional layer with a kernel size equal to 3x1. Outputs from the first and second processing sub-branches 372, 374 are combined by means of the first summation operation to obtain an output of the first processing branch. Outputs from the third and fourth processing sub-branches 376, 378 are upsampled by means of the upsample operation and then combined by means of the second summation operation to obtain an output of the second processing branch. The output of the first processing branch and the output of the second processing branch are combined by means of a concatenation operation to obtain an output of the distributed processing unit.

[0195] Figure 33 illustrates a further example of this embodiment, where the previously described processing unit 370 is used within a more complex processing unit 380 comprised in an NN- based in-loop filter, for example, within a backbone block for the architecture of NN -based in-loop filter. The distributed processing unit 370 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0196] In one embodiment, one or more distributed processing units comprised in a processing unit may comprise respective one or more scales or resolutions, where the one or more scales or resolutions may be different from each other or may be different from a scale or resolution of an input tothe processing unit or may be different from a scale or resolution of at least one component or operation comprised in the processing unit. For example, the one or more distributed processing units, or part thereof, may be applied to data with respective one or more scales or resolutions that may be different from a scale or resolution of an input to the processing unit or from a scale or resolution of at least one component or operation comprised in the processing unit.

[0197] Figure 34 illustrates an example of this embodiment in which the processing unit 390 comprises one distributed processing unit 392. The distributed processing unit comprises two processing branches, where a first processing branch gets an input tensor with Ca channels and comprises a convolutional layer with Ca kernels, and where a second processing branch gets an input tensor with Cb channels and comprises a skip connection, or identity function, or copy operation. The processing unit 390 also comprises a multiscale processing with two scales or resolutions, where the distributed processing unit 392 processes its input data at the same scale or resolution as the input to the processing unit, and where a convolutional layer “Conv, C” 394 processes data that comprises a scale or resolution that is lower than the scale or resolution of the input to the processing unit. Thus, in this example, the processing unit 390 processes its input data in two scales, where one scale is processed by a distributed processing unit 392 and another scale is processed in a non-distributed manner.

[0198] Figure 35 illustrates a further example of this embodiment, where the previously described processing unit 390 is used within a more complex processing unit comprised in an NN-based in-loop filter within, for example, a backbone block for the architecture of NN-based in-loop filter. The processing unit 390 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0199] Figure 36 illustrates another example of this embodiment that includes a processing unit 400 comprising two distributed processing units 402, 404 that process data at two different scales or resolutions. In particular, a first distributed processing unit 402 processes data at same scale as a scale of an input to the processing unit 400, and a second distributed processing unit 404 processes data at a different scale than the scale of an input to the processing unit 400. An input to the processing unit 400 is a tensor with C channels. The first distributed processing unit 402 comprises two processing branches, where a first processing branch 406 gets an input tensor with Ca channels and comprises a first convolutional layer with Ca kernels, and where a second processing branch 408 gets an input tensor with Cb channels and comprises a first skip connection, a first identity function, or a first copy operation. The second distributed processing unit 404 comprises two processing branches, where a third processing branch 410 gets an input tensor with Cc channels and comprises a second convolutional layer with Cc kernels, and where a fourth processing branch 412 gets an input tensor with Cd channels and comprises a second skip connection, second identity function, or second copy operation.

[0200] Figure 37 illustrates a further example of this embodiment, where the previously described processing unit 400 is used within a more complex processing unit comprised in an NN-basedin-loop filter, for example, within a backbone block for the architecture of NN -based in-loop filter. The distributed processing unit 350 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0201] In one embodiment, one or more distributed processing units comprised in a processing unit may comprise respective one or more scales or resolutions, where the one or more scales or resolutions may be different from each other or may be different from a scale or resolution of an input to the processing unit or may be different from a scale or resolution of at least one component or operation comprised in the processing unit, and where at least one of the one or more distributed processing units comprises two or more processing branches that process data with respective two or more directions, e.g., the two or more processing branches of the at least one of the one or more distributed processing units process their respective two or more inputs based on respective two or more different directions.

[0202] Figure 38 illustrates an example of this embodiment including a processing unit 420 comprising one distributed processing unit 422. The distributed processing unit 422 comprises two processing branches, where a first processing branch 424 gets an input tensor with Ca channels and comprises a horizontal convolutional layer with Ca kernels and a kernel size equal to IxKl, and where a second processing branch 426 gets an input tensor with Cb channels and comprises a vertical convolutional layer with Cb kernels and a kernel size equal to K2xl. The processing unit 420 also comprises a multiscale processing with two scales or resolutions, where the distributed processing unit 422 processes its input data at a different scale or resolution than a scale or resolution of the input to the processing unit 420, and where a convolutional layer “Conv, C” 428 processes data that comprises a scale or resolution that is same as the scale or resolution of the input to the processing unit 420.

[0203] Figure 39 illustrates a further example of this embodiment in which the processing unit 430 represents, for example, a backbone block comprised in an NN-based in-loop filter. The distributed processing unit 430 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0204] Figure 40 illustrates another example of this embodiment in which a distributed processing unit 440 processes data at a same scale as the scale of an input to the processing unit 442. In this embodiment, a processing unit 442 comprises one distributed processing unit 440. The distributed processing unit 440 comprises two processing branches, where a first processing branch 444 gets an input tensor with Ca channels and comprises a horizontal convolutional layer with Ca kernels and kernel size equal to 1x7, and where a second processing branch 446 gets an input tensor with Cb channels and comprises a vertical convolutional layer with Cb kernels and kernel size equal to 7x1. The processing unit 442 also comprises a multiscale processing with two scales or resolutions, where the distributed processing unit 440 processes its input data at a same scale or resolution as a scale or resolution of the input to the processing unit 442, and where a convolutional layer “Conv, C” 448processes data that comprises a scale or resolution that is different than the scale or resolution of the input to the processing unit 442.

[0205] Figure 41 illustrates a further example of this embodiment in which the processing unit 450 represents, for example, a backbone block that is comprised in an NN-based in-loop filter. The distributed processing unit 452 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0206] Figure 42 illustrates another example of this embodiment in which a first distributed processing unit 460 processes data at a same scale as the scale of an input to the processing unit 464 and a second distributed processing unit 462 processes data at a different scale than a scale of an input to the processing unit. In this embodiment, the first distributed processing unit 460 comprises a first processing branch and a second processing branch, where the first processing branch 466 gets an input tensor with Ca channels and comprises a first horizontal convolutional layer with Ca kernels and kernel size equal to IxKl, and where the second processing branch 468 gets an input tensor with Cb channels and comprises a first vertical convolutional layer with Cb kernels and kernel size equal to K2xl. The second distributed processing unit 462 comprises a third processing branch and a fourth processing branch, where the third processing branch 470 gets an input tensor with Cc channels and comprises a second horizontal convolutional layer with Cc kernels and kernel size equal to lxK3, and where the fourth processing branch 472 gets an input tensor with Cd channels and comprises a second vertical convolutional layer with Cd kernels and kernel size equal to K4xl.The processing unit 464 also comprises a multiscale processing with two scales or resolutions, where the first distributed processing unit 460 processes its input data at a same scale or resolution as a scale or resolution of the input to the processing unit 464, and where the second distributed processing unit 462 processes its input data at a scale or resolution that is lower than the scale or resolution of the input to the processing unit 464.

[0207] Figure 43 illustrates a further example of this embodiment having a processing unit 480 that represents, for a example, a backbone block that is comprised in an NN-based in-loop filter. The processing unit 480 includes other components as depicted in the prior figures and described in relation to the prior embodiments.

[0208] Some embodiments process partially overlapping subsets of channels. In one embodiment, a distributed processing unit comprises two or more processing branches where at least two of the two or more processing branches may output respective at least two intermediate outputs, where each of the at least two intermediate outputs may comprise a first subset of data and a second subset of data, and where any first subset of data in any of the at least two intermediate outputs may correspond or may be associated with a first portion of the output of the distributed processing unit, and where one second subset of data in one of the at least two intermediate outputs may correspond or may be associated with a second portion of the output of the distributed processing unit, and whereanother second subset of data in another of the at least two intermediate outputs may correspond or may be associated with a third portion of the output of the distributed processing unit. Any first subset of data in any of the at least two intermediate outputs is combined with any other first subset of data in any other of the at least two intermediate outputs by means of a first combination operation to obtain a first combined output, and the one second subset and the another second subset are combined by means of a second combination operation to obtain a second combined output. The first combined output and the second combined output are combined by means of a third combination operation to obtain an output of the distributed processing unit. Alternatively, the first combined output, the one second subset and the another second subset are combined by means of a second combination operation to obtain an output of the distributed processing unit. As used in this embodiment and related embodiments, a subset of data “corresponds” or “is associated with” a particular portion of an output when, for example, first indexes along one or more dimensions of a first tensor representing the subset of data are associated with second indexes along one or more dimensions of a second tensor representing the output based on, for example, a predetermined look-up table that specifies such association or a predetermined association rule. In one example, a first tensor corresponds to a first portion of a second tensor when at least some of the channels, or the indexes of those channels (for example, indexes from 0 to 9 along the channel dimension of a first tensor with 10 channels) in the first tensor are associated with some of the channels, or the indexes of those channels (for example, indexes from 20 to 29 along the channel dimension of a second tensor with 50 channels) in the second tensor. Figure 44 illustrates an example of this embodiment that includes a distributed processing unit 490 having a first processing branch and a second processing branch. The “Split channels” block 492 outputs a first tensor with Cc channels and a second tensor with Cd channels, where the first tensor is input to the first processing branch and the second tensor is input to the second processing branch, and where the two tensors represent two disjoint subsets of data, e.g., the first tensor and the second tensor are two disjoint subsets of an input tensor to the “Split channels” block. The first processing branch comprises a first convolutional kernel 494 with Ca kernels of size equal to KlxK2. The second processing branch comprises a second convolutional kernel 496 with Cb kernels of size equal to K3xK4. The first processing branch outputs a first output tensor with Ca channels, and the second processing branch outputs a second output tensor with Cb channels, where Ca is greater than Cc and Cb is greater than Cd. The first processing branch is configured to select and output one first subset of channels 498 based on the first output tensor. The first processing branch is also configured to select and output one second subset of channels 500 based on the first output tensor. The second processing branch is configured to select an output another first subset of channels 502 based on the second output tensor. The second processing branch is also configured to select and output another second subset of channels 504 based on the second output tensor. The second subset of the first output tensor and the second subset of the second output tensor are combined by means of a summation operation to obtain a combined subset. The combined subset, the second subset of the first output tensor and the second subsetof the second output tensor are combined by means of a concatenation operation as indicated by the Concat block 506.

[0209] In one embodiment, a distributed processing unit comprises two or more processing branches where at least two of the two or more processing branches may process respective at least two (partially) overlapping portions of an input to the distributed processing unit, and where the at least two of the two or more processing branches may output respective intermediate outputs. Each of the intermediate outputs may comprise a first subset of data and a second subset of data. Any first subset of data in any of the intermediate outputs may correspond or may be associated with a first portion of the output of the distributed processing unit, and one second subset of data in one of the intermediate outputs may correspond or may be associated with a second portion of the output of the distributed processing unit, and another second subset of data in another of the intermediate outputs may correspond or may be associated with a third portion of the output of the distributed processing unit. Any first subset of data in any of the intermediate outputs is combined with any other first subset of data in any other of the intermediate outputs by means of a first combination operation to obtain a first combined output, and the one second subset and the another second subset are combined by means of a second combination operation to obtain a second combined output. The first combined output and the second combined output are combined by means of a third combination operation to obtain an output of the distributed processing unit. Alternatively, the first combined output, the one second subset and the another second subset are combined by means of a second combination operation to obtain an output of the distributed processing unit.

[0210] Figure 45 illustrates an example of this embodiment. The processing unit comprises a distributed processing unit 510 that comprises a first processing branch and a second processing branch. The “Split channels” block 512 outputs a first tensor with Ca channels and a second tensor with Cb channels, where the first tensor and the second tensor are partially overlapping (e.g., some of the channels in the first tensor are associated with some of the channels in the second tensor), and where the first tensor is input to the first processing branch 514 and the second tensor is input to the second processing branch 516. The first processing branch 514 comprises a first convolutional kernel with Ca kernels of size equal to KlxK2. The second processing branch 516 comprises a second convolutional kernel with Cb kernels of size equal to K3xK4. The first processing branch 514 outputs a first output tensor with Ca channels, and the second processing branch 516 outputs a second output tensor with Cb channels. The first processing branch 14 is configured to select and output a first subset of channels 518 based on the first output tensor. The first processing branch 514 is also configured to select and output a second subset of channels based on the first output tensor. The second processing branch 516 is configured to select and outputs a first subset of channels 522 based on the second output tensor. The second processing branch 516 is also configured to select and output a second subset of channels 524 based on the second output tensor. The second subset of the first output tensor and the second subset of the second output tensor are combined by means of a summation operation toobtain a combined subset. The combined subset, the second subset of the first output tensor and the second subset of the second output tensor are combined by means of a concatenation operation as indicated by the Concat block 526.

[0211] In another related embodiment, the inputs of the second processing branch are downsampled before the convolution kernel on the second processing branch, and upsampled after the convolution kernel on the second processing branch.

[0212] Figures 46 and 47 illustrate examples which extend the previous two embodiments.

[0213] In another related embodiment, the inputs of the second processing branch are further split into two processing sub-branches (a first sub-branch and a second sub-branch), where the first sub-branch may select the subset of channels that are not disjoint from one another and the second sub-branch may select the disjoint subset of channels. One or more of the sub-branches may provide for downsampling before the convolution operation and upsampling after the convolution operation. The convolution operations comprised in different sub- branches may be different from each other.

[0214] Figures 48 and 49 also illustrate examples, extending on the previous two related embodiments.

[0215] As described herein, the apparatus 110 and, more particularly, the instructions 115, when executed by at least one processor 112, is configured to generate a distributed processing unit comprising a demultiplexer configured to divide the input into two or more portions and / or generate two or more copies of the input, two or more processing branches configured to differently process the respective copies of the two or more copies and / or portions of the two or more portions of the input and a combiner configured to combine the respective intermediate outputs. Additionally, the instructions, when executed by the at least one processor, define at least one or more neural network layers that are configured to serve at least a portion of the distributed processing unit, such as the two or more processing branches. As such, the performance of a device, such as an encoder and / or a decoder, that embodies the apparatus may be improved.

[0216] Figure 8 is a flowchart depicting a method according to an example embodiment. It will be understood that each block or signal and combination of blocks and signals may be implemented by various means, such as hardware, firmware, processor, circuitry, and / or other communication devices associated with execution of software including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, the computer program instructions which embody the procedures described above may be stored by the memory 114 of an apparatus 110 employing an example embodiment and executed by at least one processing circuitry 112. As will be appreciated, any such computer program instructions may be loaded onto a computer or other programmable apparatus (for example, hardware) to produce a machine, such that the resulting computer or other programmable apparatus implements the functions specified in the flowchart blocks. These computer program instructions may also bestored in a computer-readable memory that may direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture the execution of which implements the function specified in the flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart blocks.

[0217] Accordingly, blocks of the flowchart support combinations of means for performing the specified functions and combinations of operations for performing the specified functions. It will also be understood that one or more blocks of the flowchart, and combinations of blocks in the flowchart, can be implemented by special purpose hardware-based computer systems which perform the specified functions, or combinations of special purpose hardware and computer instructions.

[0218] In an embodiment, at least some of the processes described herein may be carried out by an apparatus comprising means for carrying out at least some of the described processes. Means for performing method steps as disclosed herein may include software and / or hardware components of the apparatus 110. For example, the at least one processor 112, the memory 114, and the computer program code form means for carrying out the method or methods as disclosed herein, and any of the embodiments thereof. As used herein the term “means” is to be construed in singular form, i.e. referring to a single element, or in plural form, i.e. referring to a combination of single elements. Therefore, terminology “means for [performing A, B, C]”, is to be interpreted to cover an apparatus in which there is only one means for performing A, B and C, or where there are separate means for performing A, B and C, or partially or fully overlapping means for performing A, B, C. Further, terminology “means for performing A, means for performing B, means for performing C” is to be interpreted to cover an apparatus in which there is only one means for performing A, B and C, or where there are separate means for performing A, B and C, or partially or fully overlapping means for performing A, B, C.

[0219] Even though the invention has been described above with reference to an example according to the accompanying drawings, it is clear that the invention is not restricted thereto but can be modified in several ways within the scope of the appended claims. Therefore, all words and expressions should be interpreted broadly and they are intended to illustrate, not to restrict, the embodiment. It will be obvious to a person skilled in the art that, as technology advances, the inventive concept can be implemented in various ways. Further, it is clear to a person skilled in the art that the described embodiments may, but are not required to, be combined with other embodiments in various ways.

Claims

CLAIMS1. An apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to perform at least: divide an input into two or more portions and / or generate two or more copies of the input; differently process respective copies of the two or more copies and / or portions of the two or more portions of the input to generate respective intermediate outputs; and combine the respective intermediate outputs to obtain an output or data from which the output is derivable, wherein the instructions, when executed by the at least one processor, generate a distributed processing unit comprising a demultiplexer configured to divide the input into the two or more portions and / or generate the two or more copies of the input, two or more processing branches configured to differently process the respective copies of the two or more copies and / or portions of the two or more portions of the input and a combiner configured to combine the respective intermediate outputs, and wherein the instructions, when executed by the at least one processor, define at least one or more neural network layers that are configured to serve at least as the two or more processing branches.

2. An apparatus according to Claim 1, wherein the instructions, when executed by the at least one processor, define a neural network comprising the at least one or more neural network layers that are configured to serve at least as the two or more processing branches as well as one or more previous neural network layers that are configured to generate the input to the distributed processing unit.

3. An apparatus according to Claim 1, wherein the instructions, when executed by the at least one processor, define a neural network comprising the at least one or more neural network layers that are configured to serve at least as the two or more processing branches as well as one or more following neural network layers that are configured to process the output of the distributed processing unit.

4. An apparatus according to any one of Claims 1 to 3, wherein the instructions, when executed by the at least one processor, further cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input by performing a modulation operation to multiply by one or more multiplier parameters at least one copy of the two or more copies and / or portion of the two or more portions of the input.

5. An apparatus according to any one of Claims 1 to 4, wherein the instructions, when executed by the at least one processor, further cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input by performing amodulation operation to add one or more additive parameters to at least one copy of the two or more copies and / or portion of the two or more portions of the input.

6. An apparatus according to any one of Claims 1 to 5, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit by defining a convolutional layer that has a kernel size of 1x1 and that comprises at least a portion of at least one of the two or more processing branches.

7. An apparatus according to any one of Claims 1 to 6, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit by defining a grouped convolutional layer that is configured to apply different kernels to different groups of channels of the input and that comprises at least a portion of at least one of the two or more processing branches.

8. An apparatus according to any one of Claims 1 to 7, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least two of the two or more processing branches are configured to process respective copies of the two or more copies and / or portions of the two or more portions of the input in accordance with different resolutions.

9. An apparatus according to any one of Claims 1 to 8, wherein the instructions, when executed by the at least one processor, further cause the apparatus to differently process respective copies of the two or more copies and / or portions of the two or more portions of the input by performing an identity operation.

10. An apparatus according to any one of Claims 1 to 9, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least two of the two or more processing branches are configured to process the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions.

11. An apparatus according to any one of Claims 1 to 10, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least one of thetwo or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different resolutions.

12. An apparatus according to any one of Claims 1 to 9, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different directions.

13. An apparatus according to any one of Claims 11 or 12, wherein the data processed by the at least two processing sub-branches is the same.

14. An apparatus according to any one of Claims 11 or 12, wherein the data processed by one of the at least two processing sub-branches is different than the data processed by another of the at least two processing sub-branches.

15. An apparatus according to any one of Claims 1 to 14, wherein the instructions, when executed by the at least one processor, define one or more neural network layers that are configured to serve as a distributed processing unit that is configured to process the input at a resolution different than the resolution at which the input is provided to the distributed processing unit.

16. An apparatus according to Claim 15, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least two of the two or more processing branches are configured to process the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions.

17. An apparatus according to any one of Claims 1 to 15, wherein the instructions, when executed by the at least one processor, define one or more neural network layers that are configured to serve as a distributed processing unit that is comprised of a plurality of components configured to operate at different resolutions.

18. An apparatus according to Claim 17, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that at least two of the two or more processing branches are configured to process the respective copies of the two or more copies and / or portions of the two or more portions of the input data in accordance with different directions.

19. An apparatus according to any one of Claims 1 to 18, wherein the instructions, when executed by the at least one processor, define the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit such that any processing branch is configured to generate an intermediate output comprising a first subset of data and a second subset of data, wherein the first and second subsets of data are disjoint subsets of data, wherein the first subset of data generated by any processing branch corresponds to or is associated with a first portion of an output of the distributed processing unit, and wherein the second subset of data generated by a particular processing branch corresponds to or is associated with a particular portion of an output of the distributed processing unit, and wherein the instructions, when executed by the at least one processor, define the combiner of the distributed processing unit to comprise a first combiner that is configured to combine the first subsets of data generated by the two or more processing branches.

20. An apparatus according to Claim 19, wherein the combiner further comprises a second combiner that is configured to combine the first output and the second subsets of data generated by the two or more processing branches to form a second output from which the output of the distributed processing unit is derived.

21. An apparatus according to Claim 19, wherein the combiner further comprises: a second combiner configured to combine the second subsets of data generated by the two or more processing branches to form a second output; and a third combiner configured to combine the first output and the second output to form a third output from which the output of the distributed processing unit is derived.

22. An apparatus according to Claim 19, wherein the second processing branch is defined by one or more neural network layers prior to which the respective copy of the two or more copies and / or portion of the two or more portions of the input is downsampled and after which an output of the one or more neural network layers is upsampled.

23. An apparatus according to Claim 19, wherein the second processing branch that is configured to process a respective copy of the two or more copies and / or portion of the two or more portions of the input comprised of a plurality of channels comprises a first sub-branch configured to process a subset of the plurality of channels that are associated with one another and a second sub-branch configured to process a subset of the plurality of channels that are disjoint, and wherein the first and second sub-branches are configured to comprise different neural network layers.

24. An apparatus according to any one of Claims 1 to 23, wherein the input is a tensor.

25. An apparatus according to any one of Claims 1 to 24, wherein the demultiplexer is configured to divide the input into the two or more portions and / or generate the two or more copies of the input based on one or more dimensions of the input.

26. An apparatus according to any one of Claims 1 to 25, wherein the demultiplexer is configured to divide the input into the two or more portions or generate the two or more copies of the input based on channels, spatial location or temporal location of the input.

27. An apparatus according to any one of Claims 1 to 26, wherein the two or more copies or the two or more portions into which the input is divided comprise respective two or more subsets of channels, and wherein at least two of the two or more subsets of channels are either disjoint or overlapping.

28. An apparatus according to any one of Claims 1 to 27, wherein the combiner is configured to combine the respective intermediate outputs by combining two or more tensors by one or more of concatenation, summation or stacking.

29. : An apparatus according to any of Claims 10, 12, 16 or 18, wherein the different directions comprise a vertical direction and a horizontal direction.

30. A method comprising: dividing an input into two or more portions and / or generating two or more copies of the input with a demultiplexor of a distributed processing unit; differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input with two or more processing branches of the distributed processing unit to generate respective intermediate outputs; and combining the respective intermediate outputs with a combiner of the distributed processing unit to obtain an output or data from which the output is derivable, wherein at least one or more neural network layers are configured to serve at least as the two or more processing branches.

31. A method according to Claim 30, further comprising generating the input to the distributed processing unit with one or more previous neural network layers, wherein the at least one or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more previous neural network layers.

32. A method according to Claim 30, further comprising processing the output of the distributed processing unit with one or more following neural network layers, wherein the at least one or more neural network layers that are configured to serve at least as the two or more processing branches also serve as the one or more following neural network layers.

33. A method according to any one of Claims 30 to 32, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises performing a modulation operation to multiply by one or more multiplier parameters at least one copy of the two or more copies and / or portion of the two or more portions of the input.

34. A method according to any one of Claims 30 to 33, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises performing a modulation operation to add one or more additive parameters to at least one copy of the two or more copies and / or portion of the two or more portions of the input.

35. A method according to any one of Claims 30 to 34, wherein the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit define a convolutional layer that has a kernel size of 1x1 and that comprises at least a portion of at least one of the two or more processing branches.

36. A method according to any one of Claims 30 to 35, wherein the one or more neural network layers that are configured to serve as the two or more processing branches of the distributed processing unit define a grouped convolutional layer that is configured to apply different kernels to different groups of channels of the input and that comprises at least a portion of at least one of the two or more processing branches.

37. A method according to any one of Claims 30 to 36, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises processing respective copies of the two more copies and / or portions of the two or more portions of the input in accordance with different resolutions.

38. A method according to any one of Claims 30 to 37, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises performing an identity operation.

39. A method according to any one of Claims 30 to 38, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises processing the respective copies and / or portions of the two or more portions of the input data in accordance with different directions.

40. A method according to any one of Claims 30 to 39, wherein at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different resolutions.

41. A method according to any one of Claims 30 to 39, wherein the at least one of the two or more processing branches comprises at least two processing sub-branches that are configured to process data in accordance with different directions.

42. A method according to any one of Claims 40 or 41, wherein the data processed by the at least two processing sub-branches is the same.

43. A method according to any one of Claims 40 or 41, wherein the data processed by one of the at least two processing sub-branches is different than the data processed by another of the at least two processing sub-branches.

44. A method according to any one of Claims 30 to 43, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises processing the input at a resolution different than the resolution at which the input is provided to the distributed processing unit.

45. A method according to Claim 44, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises processing the respective copies and / or portions of the two or more portions of the input data in accordance with different directions.

46. A method according to any one of Claims 30 to 45, wherein the one or more neural network layers are configured to serve as the distributed processing unit that is comprised of a plurality of components configured to operate at different resolutions.

47. A method according to Claim 46, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises processing therespective copies and / or portions of the two or more portions of the input data in accordance with different directions with at least two of the two or more processing branches.

48. A method according to any one of Claims 30 to 47, wherein differently processing respective copies of the two or more copies and / or portions of the two or more portions of the input comprises generating an intermediate output comprising a first subset of data and a second subset of data with any processing branch, wherein the first and second subsets of data are disjoint subsets of data, wherein the first subset of data generated by any processing branch corresponds to or is associated with a first portion of an output of the distributed processing unit, and wherein the second subset of data generated by a particular processing branch corresponds to or is associated with a particular portion of an output of the distributed processing unit, and wherein combining the respective intermediate outputs comprises combining the first subsets of data generated by the two or more processing branches.

49. A method according to Claim 48, further comprises combining the first output and the second subsets of data generated by the two or more processing branches to form a second output from which the output of the distributed processing unit is derived.

50. A method according to Claim 48, further comprising: combining the second subsets of data generated by the two or more processing branches to form a second output; and combining the first output and the second output to form a third output from which the output of the distributed processing unit is derived.

51. A method according to Claim 48, further comprising: downsampling the respective copy of the two or more copies and / or portion of the two or more portions of the input prior to the second processing branch; and upsampling an output of the one or more neural network layers following the second processing branch.

52. A method according to Claim 48, wherein processing, with the second processing branch, a respective copy of the two or more copies and / or portion of the of the two or more portions input comprised of a plurality of channels comprises processing a subset of the plurality of channels that are associated with one another with a first sub-branch and processing a subset of the plurality of channels that are disjoint with a second sub-branch, and wherein the first and second sub-branches are configured to comprise different neural network layers.

53. A method according to any one of Claims 30 to 52, wherein the input is a tensor.

54. A method according to any one of Claims 30 to 53, wherein generating the two or more copies of the input and / or dividing the input into the two or more portions comprises generating the two or more copies of the input and / or dividing the input into the two or more portions based on one or more dimensions of the input.

55. A method according to any one of Claims 30 to 54, wherein generating the two or more copies of the input and / or dividing the input into the two or more portions comprises generating the two or more copies of the input and / or dividing the input into the two or more portions based on channels, spatial location or temporal location of the input.

56. A method according to any one of Claims 30 to 55, wherein the two or more copies or the two or more portions into which the input is divided comprise respective two or more subsets of channels, and wherein at least two of the two or more subsets of channels are either disjoint or overlapping.

57. A method according to any one of Claims 30 to 56, wherein combining the respective intermediate outputs comprises combining two or more tensors by one or more of concatenation, summation or stacking.

58. A method according to any of Claims 39, 41, 45 or 47, wherein the different directions comprise a vertical direction and a horizontal direction.

Images