Method and apparatus for multi-branch attention with different modulation operators

Abstract

An apparatus, method, and computer program product are provided that process an input or data derivative therefrom to generate one or more attention outputs; and combine at least the input or data derivative therefrom and the one or more attention outputs to obtain an output. The apparatus, method, and computer program product also generate a multi-branch attention block comprising one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output, where the input or data derivative therefrom is input to the multi-branch attention block

Description

METHOD AND APPARATUS FOR MULTI-BRANCH ATTENTION WITH DIFFERENT MODULATION OPERATORSTECHNOLOGICAL FIELD

[0001] An apparatus, method and computer program product are provided for an improved neural network filter, and more particularly, to an improved attention mechanism for use, for example, in neural networks.BACKGROUND

[0002] A video codec consists of an encoder that transforms the input video into a compressed representation suited for storage and / or transmission and a decoder that can decompress the compressed video representation back to a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form, that is, at lower bitrate.

[0003] Typical hybrid video codecs, for example, video codecs in compliance with ITU-T H.263 and H.264, encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted, such as by motion compensation or by spatial means. Motion compensation involves finding and indicating an area in one or more previously coded video frames that corresponds closely to the block being coded, while the spatial technique uses the pixel values proximate the block to be coded in a specified manner. Secondly, the prediction error, e.g. the difference between the predicted block of pixels and the original block of pixels, is coded. This coding is typically done by transforming the difference in pixel values using a specified transform, e.g., Discrete Cosine Transform (DCT) or a variant of DCT, quantizing the resulting coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation, e.g., picture quality, and the size of the resulting coded video representation, e.g., file size or transmission bitrate.

[0004] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction, the sources of prediction are previously decoded pictures, namely, reference pictures.

[0005] In temporal inter prediction, the sources of prediction are previously decoded pictures in the same scalable layer. In intra block copy (IBC), also known as intra-block-copyprediction, prediction may be applied similarly to temporal inter prediction, but the reference picture is the current picture and only previously decoded samples can be referenced in the prediction process. Inter-layer or inter-view prediction may be applied similarly to temporal inter prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal inter prediction only, while in other cases inter prediction may refer collectively to temporal inter prediction and any of intra block copy, inter-layer prediction, and inter- view prediction provided that they are performed with the same or similar process than temporal prediction. Inter prediction, temporal inter prediction, or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0006] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in a spatial or transform domain such that either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0007] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded.Prediction of coding parameters and intra prediction may be collectively referred to as inpicture prediction.

[0008] The decoder reconstructs the output video by applying a prediction technique similar to the encoder to form a predicted representation of the pixel blocks, such as by using the motion or spatial information created by the encoder and stored in the compressed representation, and to decode the prediction error, such as by performing the inverse operation of the prediction error coding to recover the quantized prediction error signal in the spatial pixel domain. After applying prediction and prediction error decoding, the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering to improve the quality of the output video before passing the output video for display and / or storing the output video as prediction reference for the forthcoming frames in the video sequence.

[0009] In typical video codecs, the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (on the encoderside) or decoded (on the decoder side) and the prediction source block in one of the previously coded or decoded pictures. In order to represent motion vectors efficiently, the motion vectors are typically coded differentially with respect to block specific predicted motion vectors. In some video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Alternatively, motion vector predictions may be created by generating a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signalling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of a previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or or co-located blocks in a temporal reference picture. Moreover, typical high efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes a motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signalled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0010] In typical video codecs, the prediction residual after motion compensation is first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform that can, in many cases, help reduce this correlation and provide more efficient coding.

[0011] Some video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g. the desired macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor I to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:C = D + XRwhere C is the Lagrangian cost to be minimized, D is the image distortion, e.g., mean squared error, with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0012] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or the like. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units. A prefix SEI NAL unit can start a picture unit or the like and a suffix SEI NAL unit can end a picture unit or the like. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in the H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards, and the user data SEI messages enable entities to specify SEI messages for their own use. The standards may contain the syntax and semantics for the specified SEI messages, but a process for handling the messages by the recipient may not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when the encoders create SEI message(s), but decoders may not be required to process SEI messages for output order conformance. By including the syntax and semantics of SEI messages in standards allows different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.

[0013] While encoders and decoders are effective, their encoding and decoding performance and, in turn, the resulting video storage and / or transmission experience may be still be desirably improved.BRIEF SUMMARY

[0014] An apparatus, method and computer program product are provided for an improved neural network filter.

[0015] According to an aspect of the present disclosure, there is provided an apparatus including at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: process an input or data derivative therefrom to generate one or more attention outputs and combine at least the input or data derivative therefrom and the one or more attention outputs to obtain an output. The apparatus is also caused to generate a multi-branch attention block including one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least theinput or data derivative therefrom and the one or more attention outputs to generate the output. The input or data derivative therefrom is input to the multi-branch attention block.

[0016] According to some embodiments, the input or data derivative therefrom that is input to the multi-branch attention block is an input tensor, the output is an output tensor, and the one or more attention outputs are attention tensors. According to some embodiments, at least one attention branch of the one or more attention branches includes at least one aggregation operation configured to generate an aggregation output that represents an aggregated version of an input to the at least one aggregation operation. In this embodiment, the input to the at least one aggregation operation is the input or data derivative therefrom. At least one aggregation operation of some embodiments receives an input tensor and is configured to reduce a size of one or more dimensions of the input tensor or reduce a height and a width of the input tensor or reduce a number of channels of the input tensor. The at least one aggregation operation of some embodiments includes one or more of a mean operation, average operation, summation operation, median operation, mode operation, minimum operation, or maximum operation.

[0017] According to some embodiments, at least one attention branch of the one or more attention branches includes at least one or more neural network layers. The output of some embodiments is an output of the multi-branch attention block and includes at least one attention output of the one or more attention outputs. The apparatus of some embodiments is caused to generate a second multi-branch attention block that is configured to receive the output including the at least one attention output. In this embodiment, the second multibranch attention block includes at least one attention branch configured to skip at least one aggregation operation by using the at least one attention output as an aggregation output.

[0018] According to some embodiments the attention usage block includes at least one or more modulation operations configured to modulate or modify a first signal based on a second signal to obtain a modulation output. In this embodiment, the first signal includes at least the input or data derivative therefrom that is input to the multi-branch attention block and the second signal includes at least one or more of the one or more attention outputs. The apparatus of some embodiments is caused to combine at least two or more modulation outputs of respective two or more modulation operations and the input or data derivative therefrom via a combination operation to obtain a combined output. In this embodiment, the combined output is included in the output. According to some embodiments, a modulation operation of the at least one or more modulation operations includes an element wise multiplication or an element wise summation. At least two modulation operations of someembodiments are the same. At least two modulation operations of other embodiments are different. In an example embodiment, an input to at least one modulation operation or an output of at least one modulation operation includes a tensor. In another example embodiment, two or more inputs to the respective two or more modulation operations or the combination operation include two or more tensors with different shapes. In this embodiment, the apparatus is further caused to perform a broadcasting operation in one or more dimensions or axes of the two or more tensors.

[0019] According to some embodiments, the attention usage block includes a first modulation operation and a second modulation operation. In this embodiment, the multibranch attention block includes at least: a first attention branch including at least a first aggregation operation and one or more first neural network layers. The multi-branch attention block further includes a second attention branch including at least a second aggregation operation and one or more second neural network layers. The first aggregation operation is configured to receive the input or data derivative therefrom and output a first aggregation output. The second aggregation operation is configured to receive the input or data derivative therefrom and output a second aggregation output. The one or more first neural network layers are configured to receive the first aggregation output and output a first attention output. The one or more second neural network layers are configured to receive the second aggregation output and output a second attention output. The first modulation operation is configured to receive the first attention output and the input or data derivative therefrom and output a first modulation output. The second modulation operation is configured to receive the first modulation output, the second attention output, and the input or data derivative therefrom and output the output of the multi-branch attention block. The first aggregation operation includes a first mean operation, the second aggregation operation includes a second mean operation, the one or more first neural network layers include a first convolutional layer, the one or more second neural network layers include a second convolutional layer, the first modulation operation includes a multiplication operation and the second modulation operation includes a summation operation.

[0020] According to another aspect of the present disclosure, a method is provided that includes processing an input or data derivative therefrom to generate one or more attention outputs and combining at least the input or data derivative therefrom and the one or more attention outputs to obtain an output. The method of an example embodiment further includes generating a multi-branch attention block including one or more attention branches configured to process the input or data derivative therefrom to generate the one or moreattention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output. The input or data derivative therefrom is input to the multi-branch attention block.

[0021] According to some embodiments, the input or data derivative therefrom that is input to the multi-branch attention block is an input tensor, the output is an output tensor, and the one or more attention outputs are attention tensors. According to some embodiments, at least one attention branch of the one or more attention branches includes at least one aggregation operation configured to generate an aggregation output that represents an aggregated version of an input to the at least one aggregation operation. In this embodiment, the input to the at least one aggregation operation is the input or data derivative therefrom. At least one aggregation operation of some embodiments receives an input tensor and is configured to reduce a size of one or more dimensions of the input tensor or reduce a height and a width of the input tensor or reduce a number of channels of the input tensor. The at least one aggregation operation of some embodiments includes one or more of a mean operation, average operation, summation operation, median operation, mode operation, minimum operation, or maximum operation.

[0022] According to some embodiments, at least one attention branch of the one or more attention branches includes at least one or more neural network layers. The output of some embodiments is an output of the multi-branch attention block and includes at least one attention output of the one or more attention outputs. The method of some embodiments further includes generating a second multi-branch attention block that is configured to receive the output including the at least one attention output. In this embodiment, the second multibranch attention block includes at least one attention branch configured to skip at least one aggregation operation by using the at least one attention output as an aggregation output.

[0023] According to some embodiments the attention usage block includes at least one or more modulation operations configured to modulate or modify a first signal based on a second signal to obtain a modulation output. In this embodiment, the first signal includes at least the input or data derivative therefrom that is input to the multi-branch attention block and the second signal includes at least one or more of the one or more attention outputs. The method of some embodiments further includes combining at least two or more modulation outputs of respective two or more modulation operations and the input or data derivative therefrom via a combination operation to obtain a combined output. In this embodiment, the combined output is included in the output. According to some embodiments, a modulation operation of the at least one or more modulation operations includes an element wisemultiplication or an element wise summation. At least two modulation operations of some embodiments are the same. At least two modulation operations of other embodiments are different. In an example embodiment, an input to at least one modulation operation or an output of at least one modulation operation includes a tensor. In another example embodiment, two or more inputs to the respective two or more modulation operations or the combination operation include two or more tensors with different shapes. In this embodiment, the method further includes performing a broadcasting operation in one or more dimensions or axes of the two or more tensors.

[0024] According to some embodiments, the attention usage block includes a first modulation operation and a second modulation operation. In this embodiment, the multibranch attention block includes at least: a first attention branch including at least a first aggregation operation and one or more first neural network layers. The multi-branch attention block further includes a second attention branch including at least a second aggregation operation and one or more second neural network layers. The first aggregation operation is configured to receive the input or data derivative therefrom and output a first aggregation output. The second aggregation operation is configured to receive the input or data derivative therefrom and output a second aggregation output. The one or more first neural network layers are configured to receive the first aggregation output and output a first attention output. The one or more second neural network layers are configured to receive the second aggregation output and output a second attention output. The first modulation operation is configured to receive the first attention output and the input or data derivative therefrom and output a first modulation output. The second modulation operation is configured to receive the first modulation output, the second attention output, and the input or data derivative therefrom and output the output of the multi-branch attention block. The first aggregation operation includes a first mean operation, the second aggregation operation includes a second mean operation, the one or more first neural network layers include a first convolutional layer, the one or more second neural network layers include a second convolutional layer, the first modulation operation includes a multiplication operation and the second modulation operation includes a summation operation.

[0025] According to another aspect of the present disclosure, there is provided a computer program product, including at least one non-transitory computer-readable storage medium having computer-executable program code portions stored therein with the computer-executable program code portions comprising program code instructions configured to: process an input or data derivative therefrom to generate one or more attentionoutputs and combine at least the input or data derivative therefrom and the one or more attention outputs to obtain an output. The program code instructions are also configured to generate a multi-branch attention block including one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output. The input or data derivative therefrom is input to the multi-branch attention block.

[0026] According to some embodiments, the input or data derivative therefrom that is input to the multi-branch attention block is an input tensor, the output is an output tensor, and the one or more attention outputs are attention tensors. According to some embodiments, at least one attention branch of the one or more attention branches includes at least one aggregation operation configured to generate an aggregation output that represents an aggregated version of an input to the at least one aggregation operation. In this embodiment, the input to the at least one aggregation operation is the input or data derivative therefrom. At least one aggregation operation of some embodiments receives an input tensor and is configured to reduce a size of one or more dimensions of the input tensor or reduce a height and a width of the input tensor or reduce a number of channels of the input tensor. The at least one aggregation operation of some embodiments includes one or more of a mean operation, average operation, summation operation, median operation, mode operation, minimum operation, or maximum operation.

[0027] According to some embodiments, at least one attention branch of the one or more attention branches includes at least one or more neural network layers. The output of some embodiments is an output of the multi-branch attention block and includes at least one attention output of the one or more attention outputs. The program code instructions of some embodiments are also configured to generate a second multi-branch attention block that is configured to receive the output including the at least one attention output. In this embodiment, the second multi-branch attention block includes at least one attention branch configured to skip at least one aggregation operation by using the at least one attention output as an aggregation output.

[0028] According to some embodiments the attention usage block includes at least one or more modulation operations configured to modulate or modify a first signal based on a second signal to obtain a modulation output. In this embodiment, the first signal includes at least the input or data derivative therefrom that is input to the multi-branch attention block and the second signal includes at least one or more of the one or more attention outputs. Theprogram code instructions of some embodiments are also configured to combine at least two or more modulation outputs of respective two or more modulation operations and the input or data derivative therefrom via a combination operation to obtain a combined output. In this embodiment, the combined output is included in the output. According to some embodiments, a modulation operation of the at least one or more modulation operations includes an element wise multiplication or an element wise summation. At least two modulation operations of some embodiments are the same. At least two modulation operations of other embodiments are different. In an example embodiment, an input to at least one modulation operation or an output of at least one modulation operation includes a tensor. In another example embodiment, two or more inputs to the respective two or more modulation operations or the combination operation include two or more tensors with different shapes. In this embodiment, the program code instruction are also configured to perform a broadcasting operation in one or more dimensions or axes of the two or more tensors.

[0029] According to some embodiments, the attention usage block includes a first modulation operation and a second modulation operation. In this embodiment, the multibranch attention block includes at least: a first attention branch including at least a first aggregation operation and one or more first neural network layers. The multi-branch attention block further includes a second attention branch including at least a second aggregation operation and one or more second neural network layers. The first aggregation operation is configured to receive the input or data derivative therefrom and output a first aggregation output. The second aggregation operation is configured to receive the input or data derivative therefrom and output a second aggregation output. The one or more first neural network layers are configured to receive the first aggregation output and output a first attention output. The one or more second neural network layers are configured to receive the second aggregation output and output a second attention output. The first modulation operation is configured to receive the first attention output and the input or data derivative therefrom and output a first modulation output. The second modulation operation is configured to receive the first modulation output, the second attention output, and the input or data derivative therefrom and output the output of the multi-branch attention block. The first aggregation operation includes a first mean operation, the second aggregation operation includes a second mean operation, the one or more first neural network layers include a first convolutional layer, the one or more second neural network layers include a second convolutional layer, thefirst modulation operation includes a multiplication operation and the second modulation operation includes a summation operation.

[0030] According to another aspect of the present disclosure, there is provided an apparatus including means for processing an input or data derivative therefrom to generate one or more attention outputs and means for combining at least the input or data derivative therefrom and the one or more attention outputs to obtain an output. The apparatus also includes means for generating a multi-branch attention block including one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output. The input or data derivative therefrom is input to the multi-branch attention block.

[0031] According to some embodiments, the input or data derivative therefrom that is input to the multi-branch attention block is an input tensor, the output is an output tensor, and the one or more attention outputs are attention tensors. According to some embodiments, at least one attention branch of the one or more attention branches includes at least one aggregation operation configured to generate an aggregation output that represents an aggregated version of an input to the at least one aggregation operation. In this embodiment, the input to the at least one aggregation operation is the input or data derivative therefrom. At least one aggregation operation of some embodiments receives an input tensor and is configured to reduce a size of one or more dimensions of the input tensor or reduce a height and a width of the input tensor or reduce a number of channels of the input tensor. The at least one aggregation operation of some embodiments includes one or more of a mean operation, average operation, summation operation, median operation, mode operation, minimum operation, or maximum operation.

[0032] According to some embodiments, at least one attention branch of the one or more attention branches includes at least one or more neural network layers. The output of some embodiments is an output of the multi-branch attention block and includes at least one attention output of the one or more attention outputs. The apparatus further includes means for generating a second multi-branch attention block that is configured to receive the output including the at least one attention output. In this embodiment, the second multi-branch attention block includes at least one attention branch configured to skip at least one aggregation operation by using the at least one attention output as an aggregation output.

[0033] According to some embodiments the attention usage block includes at least one or more modulation operations configured to modulate or modify a first signal based on asecond signal to obtain a modulation output. In this embodiment, the first signal includes at least the input or data derivative therefrom that is input to the multi-branch attention block and the second signal includes at least one or more of the one or more attention outputs. The apparatus further includes means for combining at least two or more modulation outputs of respective two or more modulation operations and the input or data derivative therefrom via a combination operation to obtain a combined output. In this embodiment, the combined output is included in the output. According to some embodiments, a modulation operation of the at least one or more modulation operations includes an element wise multiplication or an element wise summation. At least two modulation operations of some embodiments are the same. At least two modulation operations of other embodiments are different. In an example embodiment, an input to at least one modulation operation or an output of at least one modulation operation includes a tensor. In another example embodiment, two or more inputs to the respective two or more modulation operations or the combination operation include two or more tensors with different shapes. In this embodiment, the apparatus further includes means for performing a broadcasting operation in one or more dimensions or axes of the two or more tensors.

[0034] According to some embodiments, the attention usage block includes a first modulation operation and a second modulation operation. In this embodiment, the multibranch attention block includes at least: a first attention branch including at least a first aggregation operation and one or more first neural network layers. The multi-branch attention block further includes a second attention branch including at least a second aggregation operation and one or more second neural network layers. The first aggregation operation is configured to receive the input or data derivative therefrom and output a first aggregation output. The second aggregation operation is configured to receive the input or data derivative therefrom and output a second aggregation output. The one or more first neural network layers are configured to receive the first aggregation output and output a first attention output. The one or more second neural network layers are configured to receive the second aggregation output and output a second attention output. The first modulation operation is configured to receive the first attention output and the input or data derivative therefrom and output a first modulation output. The second modulation operation is configured to receive the first modulation output, the second attention output, and the input or data derivative therefrom and output the output of the multi-branch attention block. The first aggregation operation includes a first mean operation, the second aggregation operation includes a second mean operation, the one or more first neural network layers include a first convolutionallayer, the one or more second neural network layers include a second convolutional layer, the first modulation operation includes a multiplication operation and the second modulation operation includes a summation operation.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Having thus described certain example embodiments of the present disclosure in general terms, reference will hereinafter be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:

[0036] Figure 1 illustrates a system pipeline for video coding for machines;

[0037] Figure 2 illustrates a neural network based in-loop filter;

[0038] Figure 3 illustrates an apparatus that is capable of being configured in accordance with an example embodiment.

[0039] Figure 4 illustrates the operations performed, such as by the apparatus of Figure 3, in accordance with an example embodiment;

[0040] Figure 5 illustrates a multi-branch attention block in accordance with at least an example embodiment;

[0041] Figure 6 illustrates a multi-branch attention block in which at least one attention branch includes an aggregation operation in accordance with at least an example embodiment;

[0042] Figure 7 illustrates a multi-branch attention block in accordance with other example embodiments;

[0043] Figure 8 illustrates a multi-branch attention block in which at least one attention branch includes an aggregation operation and one or more neural network layers in accordance with at least an example embodiment;

[0044] Figure 9 illustrates a multi-branch attention block in accordance with another example embodiment;

[0045] Figure 10 illustrates a multi-branch attention block in which at least one attention output is output by the multi-branch attention block in accordance with at least an example embodiment;

[0046] Figure 11 illustrates a multi-branch attention block in which an output of the multi-branch attention block is input to a second multi-branch attention block in accordance with at least an example embodiment;

[0047] Figure 12 illustrates a multi-branch attention block in which at least two modulation outputs are combined by means of a combination operation in accordance with at least an example embodiment;

[0048] Figure 13 illustrates a multi-branch attention block in accordance with at least some example embodiments;

[0049] Figure 14 illustrates a multi-branch attention block in accordance with another example embodiment;

[0050] Figure 15 illustrates a multi-branch attention block in accordance with further example embodiments;

[0051] Figure 16 illustrates a multi-branch attention block in accordance with at least an example embodiment;

[0052] Figure 17 illustrates a multi-branch attention block in accordance with another example embodiment;

[0053] Figure 18 illustrates a multi-branch attention block in accordance with other example embodiments;

[0054] Figure 19 illustrates a multi-branch attention block in accordance with at least an example embodiment;

[0055] Figure 20 illustrates a multi-branch attention block in accordance with another example embodiment; and

[0056] Figure 21 illustrates a multi-branch attention block in accordance with further example embodiments.DETAILED DESCRIPTION

[0057] The following embodiments are exemplary. Although the specification may refer to “an”, “one”, or “some” embodiment(s) in several locations of the text, this does not necessarily mean that each reference is made to the same embodiment(s), or that a particular feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments. Further, when a particular feature, structure, or characteristic is described in connection of an embodiment, it is within the knowledge of one skilled in the art to apply such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. It shall be understood that although the terms “first,” “second” and the like may be used herein to describe various elements,these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0058] For the purposes of the present disclosure, the phrases “at least one of A or B”, “at least one of A and B”, and “A and / or B” means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0059] Neural networks (NNs) may be used in the context of image and video compression in order to improve the encoding and decoding performance. A neural network may be described as a computation graph consisting of several layers of computation. Each layer may consist of one or more units with each unit performing an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, e.g., values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers. As used herein, the terms “model”, “neural network”, “neural net” and “network” are interchangeable. In addition, the weights of neural networks are sometimes referred to as learnable parameters or simply as parameters.

[0060] Two of the most widely used architectures for neural networks are feed-forward and recurrent architectures. Feed-forward neural networks are such that there is no feedback loop: each layer takes input from one or more of the layers before and provides its output as the input for one or more of the subsequent layers. Additionally, units inside a certain layer may take input from units in one or more of preceding layers and provide output to one or more of following layers.

[0061] Initial layers close to the input data extract semantically low-level features such as edges and textures in images, and intermediate and final layers extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc. In recurrent neural nets, there is a feedback loop, so that the network becomes stateful, e.g., the network is able to memorize information or a state.

[0062] Neural networks are being utilized in an ever-increasing number of applications for many different types of device, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc.

[0063] The most important property of neural nets (and other machine learning tools) is that they are able to learn properties from input data, either in supervised manner or inunsupervised manner. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal.

[0064] In general, the training algorithm consists of changing some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class or category index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output’s error, also referred to as the loss. Examples of losses are mean squared error (MSE), crossentropy, etc. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural net to make a gradual improvement of the network’s output, e.g., to gradually decrease the loss.

[0065] Training a neural network is an optimization process, but the final goal is different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, e.g., data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, e.g., to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to determine if the network is learning. If the neural network is learning, the training set error should decrease. Otherwise, the neural network is in the regime of underfitting. The errors are also monitored during the training process to determine if the network is learning to generalize. If the neural network is learning to generalize, the validation set error decreases and is not much higher than the training set error. However, if the training set error is low, but the validation set error is much higher than the training set error, or the validation set error does not decrease, or the validation set error even increases, the neural network may be in the regime of overfitting. The regime of overfitting means that the neural network has just memorized the properties of the training set and performs well only on that set, but performs poorly on a set not used for tuning its parameters.

[0066] Lately, neural networks have been used for compressing and de-compressing data such as images, e.g., in an image codec. The most widely used architecture for realizing one component of an image codec is the auto-encoder, which is a neural network consisting of two parts: a neural encoder and a neural decoder. The encoder takes as input an image and produces a code which requires less bits than the input image. This code may be obtained by applying a binarization or quantization process to the output of the encoder. The decoder takes in this code and reconstructs the image which was input to the encoder.

[0067] Such encoder and decoder are usually trained to minimize a combination of bitrate and distortion, where the distortion may be based on one or more of the following metrics: Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), or similar. These metrics are meant to be correlated to the human visual perception quality, so that minimizing or maximizing one or more of these metrics results into improving the visual quality of the decoded image as perceived by humans.

[0068] As used herein, the terms, “neural encoder” and “neural decoder” may be referred to as “encoder” and “decoder” respectively, even though the use of such terms may refer to algorithms which are learned from data instead of being tuned by hand.

[0069] Reducing the distortion in image and video compression is often intended to increase human perceptual quality, as humans are considered to be the end users, e.g., consuming / watching the decoded image. Recently, with the advent of machine learning, especially deep learning, there is a rising number of machines (e.g., autonomous agents) that analyze data independently from humans and that may even take decisions based on the analysis results without human intervention. Examples of such analysis include object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. Example use cases and applications include self-driving cars, video surveillance cameras and public safety, smart sensor networks, smart TV and smart advertisement, person re-identification, smart traffic monitoring, drones, etc. This may raise the following question: when decoded data is consumed by machines, should a different quality metric be used -other than human perceptual quality- when considering media compression in inter-machine communications? Additionally, dedicated algorithms for compressing and decompressing data for machine consumption are likely to be different than those for compressing and decompressing data for human consumption. The set of tools and concepts for compressing and decompressing data for machine consumption is referred to herein as “Video Coding for Machines”.

[0070] In the context of Video Coding for Machines, it is likely that the receiver-side device has multiple “machines” or neural networks (NNs). These multiple machines may be used in a certain combination which is for example determined by an orchestrator subsystem. The multiple machines may be used for example in succession, based on the output of the previously used machine, and / or in parallel. For example, a video which was compressed and then decompressed may be analyzed by one machine (NN) for detecting pedestrians, by another machine (another NN) for detecting cars, and by another machine (another NN) for estimating the depth of all the pixels in the frames.

[0071] As used herein, the terms “receiver-side” or “decoder-side” may be used to refer to the physical or abstract entity or device which contains one or more machines, and runs these one or more machines on some encoded and eventually decoded video representation which is encoded by another physical or abstract entity or device, such as the “encoder-side device”.

[0072] The encoded video data may be stored into a memory device, for example as a file. The stored file may later be provided to another device. Alternatively, the encoded video data may be streamed from one device to another.

[0073] FIG. 1 illustrates a general pipeline of Video Coding for Machines (VCM). A VCM encoder 102 encodes the input video into a bitstream. A bitrate may be computed from the bitstream in order to evaluate the size of the bitstream. A VCM decoder 104 decodes the bitstream output by the VCM encoder. The output of the VCM decoder 104 is referred in FIG. 1 as “Decoded data for machines”. This data may be considered as the decoded or reconstructed video. However, in some implementations of this pipeline, this data may not have same or similar characteristics as the original video which was input to the VCM encoder 102. For example, this data may not be easily understandable by a human by simply rendering the data onto a screen. The output of VCM decoder 104 is then input to one or more task neural networks 106. In FIG. 1, for the sake of illustrating that there may be any number of task-NNs 106, there are three example task-NNs, and a non-specified one (Task-NN X). One goal of VCM may be to obtain a low bitrate while guaranteeing that the task-NNs 16 still perform well in terms of the evaluation metric associated to each task.

[0074] When a conventional video encoder, such as a H.266 / VVC encoder, is used as a VCM encoder 102, any one of several approaches may be used to adapt the encoding to be suitable to machine analysis tasks. For example, a filter is used to preprocess the input to a conventional encoder. The filter may be a machine learning based filter, such as a convolutional neural network. In another example, one or more regions of interest (ROIs)may be detected using an ROI detection method. In this regard, ROI detection may be performed using a task NN, such as an object detection NN. In some cases, ROI boundaries of a group of pictures or an intra period may be spatially overlaid and rectangular areas may be formed to cover the ROI boundaries. The detected ROIs, such as rectangular areas, may be used in one or more ways. For example, the quantization parameter (QP) may be adjusted spatially in a manner that ROIs are encoded using finer quantization step size(s) than other regions. For example, the QP may be adjusted CTU-wise. As other examples, the video may be preprocessed to contain only the ROIs, while the other areas are replaced by one or more constant values or removed. In another example, a grid is formed in a manner that a single grid cell covers a ROI. Grid rows or grid columns that contain no ROIs are downsampled as preprocessing to encoding. In a further example, a quantization parameter of the highest temporal sublayer(s) is increased, e.g., coarser quantization is used, when compared to practices for human watchable video. In an additional example, the original video is temporally downsampled as preprocessing prior to encoding. A frame rate upsampling method may be used as postprocessing subsequent to decoding, if machine analysis at the original frame rate is desired.

[0075] A neural network may be used for filtering or processing input data. In this instance, the neural network may be referenced as a neural network based filter, an NN filter or a filter. An NN filter may include one or more neural networks, and optionally one or more components that may not be categorized as neural networks. An NN filter may perform various function including, but may not be limited to, visual enhancement, colourization, upsampling, super-resolution, inpainting, temporal extrapolation, generating content, or the like.

[0076] In some video codecs, a neural network may be used as filter in the encoding and decoding loop (also referred to simply as coding loop). In this instance, the neural network may be referred to as a neural network loop filter, or a neural network in-loop filter. The NN loop filter may replace all other loop filters of an existing video codec, or may represent an additional loop filter with respect to the already present loop filters in an existing video codec.

[0077] In one example, a codec is a modified VVC / H.266 compliant codec (e.g., a VVC / H.266 compliant codec that has been modified and, as a result, may no longer be compliant to the VVC / H.266 standard). In this example, the codec includes one or more NN loop filters. An input to the one or more NN loop filters may include at least a reconstructed block or frames (simply referred to as reconstruction) or data derived from a reconstructedblock or frame, e.g., the output of a conventional loop filter. The reconstruction may be obtained based on predicting a block or frame, such as by intra-frame prediction or interframe prediction, and performing residual compensation. The one or more NN loop filters may enhance the quality of at least one of their input, so that a rate-distortion loss is decreased. The rate may indicate a bitrate (estimated or real) of the encoded video. The distortion may indicate a pixel fidelity distortion, such as mean-squared error (MSE), mean absolute error (MAE), mean average precision (mAP) computed based on the output of a task NN, such as an object detection NN, when the input is the output of the post-processing NN, or other machine task-related metric for tasks such as object tracking, video activity classification, video anomaly detection, etc. The enhancement may result into a coding gain, which can be expressed for example in terms of a Bjontegaard delta (BD) rate or a BD-peak signal to noise ratio (PSNR).

[0078] A neural network filter may be used as post-processing filter for a codec, such as by being applied to an output of an image or video decoder in order to remove or reduce coding artifacts. In one example, the NN filter is used as a post-processing filter in which the input includes data that is output by or is derived from an output of a traditional decoder, such as a decoder that is compliant with the VVC / H.266 standard. In another example, the NN filter is used as a post-processing filter in which the input includes data that is output by or is derived from an output of a decoder of an end-to-end learned decoder.

[0079] In the case of filtering images, an NN filter may take as input at least one or more first images to be filtered and may output at least one or more second images. The one or more second images are the filtered version of the one or more first images. In one example, the filter takes as input one image and outputs one image. In another example, the filter takes as input more than one image and outputs one image. In another example, the filter takes as input more than one image and outputs more than one image.

[0080] An NN filter may take as input also other data (also referred to as auxiliary data, or extra data) than the data that is to be filtered. The other data may be data that can aid the filter to perform a better filtering than if no auxiliary data was provided as input. In one example, the auxiliary data includes information about prediction data, and / or information about the picture type, and / or information about the slice type, and / or information about a quantization parameter (QP) used for encoding, and / or information about boundary strength, etc. In one example, the filter takes as input one image and other data associated with that image, such as information about the quantization parameter (QP) used for quantizing and / or dequantizing that image, and outputs one image.

[0081] An example architecture of a neural network based in-loop filter that is included in a video decoder is depicted in FIG. 2. In this example, “luma” and “chroma” refer to the reconstructed luma and chroma, respectively, that are to be enhanced by the NN filter, and may represent an intermediate result of an encoding or decoding operation. In one example in which the filter is a loop filter of a video codec, “luma” and “chroma” may represent the result of combining a predicted block with a decoded residual. The luma and chroma, that may be extracted from a YUV420 signal, are provided as input to a Discrete Cosine Transform (DCT), obtaining DCT-transformed luma and DCT-transformed chroma that are then concatenated to form the reconstruction Rec.

[0082] The terms Rec, Pred, BS, BaseQP, SliceQP and IPB represent the inputs to the NN filter and each of those inputs is usually in the format of a tensor of shape BxCxHxW in which B indicates a batch size, e.g., number of pictures or blocks, C indicates a number of channels, and H and W indicate a height and width, respectively. The square brackets and the number within them, e.g., Rec[3], indicate the number of channels of the associated tensor. For example, Rec[3] indicates that the input tensor Rec contains 3 channels and thus has shape Bx3xHxW, where the three channels may represent the luma channel, the Blue-Yellow Chrominance (Cb) channel and the Red-Green Chrominance (Cr) channel. The Cb channel and the Cr channel may be collectively referred to as chroma.

[0083] Pred refers to prediction. BS refers to boundary strength, BaseQP refers to the sequence-level quantization parameter (QP), SliceQP refers to the slice-level QP and IPB refers to the type of slice or type of picture, e.g., intra slice, P inter slice or B inter slice). In some embodiments, Rec may be referred to as main input, or data to be filtered, whereas Pred, BS, BaseQP, SliceQP and IPB may be referred to as auxiliary input, or data not to be filtered.

[0084] Each block in Figure 2 represents an operation, such as one or more NN layers. By way of example, the block “ConvKlxK2,Z” indicates a convolutional layer with kernel size KlxK2 and a number of kernels equal to Z. When present, the term “+ PReLU” indicates that a layer is followed by a Parameterized Rectified Linear Unit (PReLU). When present, the term “PReLU +” indicates that a layer is preceded by a Parameterized Rectified Linear Unit (PReLU). When present, the term “s=2” indicates that a convolutional layer has stride equal to 2; when not present, the convolutional layer has stride equal to 1. “Split” refers to an operation that splits a tensor across the channel dimension.

[0085] ‘Luma BB” 202 and “Chroma BB” 204 indicate a backbone block used for filtering or processing the luma channel and the chroma channel, respectively. Thearchitecture of a backbone block 206 is also shown in Figure 2 and includes several layers and operations. “SepConv3x3” indicates a block 208 that comprises a separable convolution. An illustration of the SepConv3x3 block 208 is also shown in Figure 2 and includes several layers. “PS” refers to a Pixel Shuffle operation, which rearranges elements in a tensor of shape Bx(C*r*r)xHxW to a tensor of shape BxCx(H*r)x(W*r), where r is an upscale factor. “LumaOut” and “ChromaOut” represent the filtered luma and the filtered chroma, respectively, e.g., the final outputs from the NN filter.

[0086] By way of illustration, the NN architecture is figuratively organized into the following sections: head, fuse, transition, luma backbone, chroma backbone, luma tail 210 and chroma tail 212. However, it is to be noted that other possible organizations of the NN into subsets or blocks or sections may be possible and may still be in the scope of the present disclosure.

[0087] The input tensors are input to respective convolutional layers that are part of the “head” section of the NN filter. The outputs of those convolutional layers are tensors, referred to head tensors. As part of the operations of the “fuse” section of the NN, the head tensors are concatenated into a single tensor across the channel dimension. It is noted, however, that the concatenation operation is not illustrated. The concatenated tensor is input to a convolutional layer, followed by a non-linear activation function PReLU. The output of the fuse section is input to the “transition” section of the NN, which includes a separable convolutional layer with stride equal to 2 and a convolutional layer and a PReLU layer. The output of the transition section is a tensor of shape Bx(2*C)x(H / 2)x(W / 2), and the tensor is split into two sub-tensors, where each subtensor is of shape BxCx(H / 2)x(W / 2). A first subtensor is used to filter the luma and a second subtensor is used to filter the chroma. The first subtensor is input to a convolutional layer that maps the number of channel C to a different number of channels, followed the “luma backbone” section 202, and the second subtensor is input to a convolutional layer that maps the number of channel C to a different number of channels, followed by the “chroma backbone” section 204. The luma backbone section 202 includes Ny luma backbone blocks, and the chroma backbone section 204 includes Nuv chroma backbone blocks. The output of a backbone block is input to the next backbone block, until the last backbone block in the section. The output of the last luma backbone block is input to the “luma tail” section 210, that includes a 1x1 convolutional layer, a separable convolution SepConv3x3, another 1x1 convolutional layer and a PReLU operation, a Conv3x3 layer, a Pixel Shuffle (PS) operation, and an inverse DCT operation. The output of the PixelShuffle operation is added to the input luma, in order to obtain LumaOut. The output of the lastchroma backbone block is input to the “chroma tail” section 212, that includes a 1x1 convolutional layer, a separable convolution SepConv3x3, another 1x1 convolutional layer and a PReLU operation, a Conv3x3 layer and an inverse DCT operation. The output of the inverse DCT operation is added to the input chroma, in order to obtain ChromaOut.

[0088] FIG. 2 also provides examples of the values of various hyper-parameters of the NN, such as the number of channels of convolutional layers designated DI, D2, D3, D4, D5, D6, C and Cl, and the number of luma and chroma backbone blocks Ny and Nuv.

[0089] As used herein, the terms frame, picture and image are used interchangeably. The input and output to an end-to-end learned codec may be any type of content, such as audio or video, e.g., pictures. As such, the input and output of a NN filter may be pictures. In some embodiments, when the term block refers to a portion of a picture, the block may alternatively be referred to as a frame, a picture or image. In other words, at least some embodiments, even when described as applied to a picture, may be applicable also to a block, e.g., to a portion of a picture.

[0090] As noted above, neural networks are increasingly present in an encode and / or in a decoder, such as in a video encoder and / or a video decoder. One example is a NN in-loop filter. Other examples include a NN post-processing filter or the NN may be used as part of an intra-frame prediction process, an inter-frame prediction process, a transformation process such as a transformation of a prediction residual signal, an end-to-end learned codec such as a NN that takes a lossless-decoded latent tensor and outputs reconstructed or decoded data. The NN may also perform spatial upsampling and / or super-resolution. An apparatus, method and computer program product are provided to improve the performance of such neural networks. For example, various embodiments described herein address filtering an input data item, such as an image or a video frame, for one or more purposes including, but not limited to, enhancing visual quality, enhancing machine analysis results, etc. More specifically, various embodiments described herein address the architecture of a NN filter, for example, improving the attention mechanism of neural networks.

[0091] By way of example, but not of limitation, the apparatus, method and computer program product of an example embodiment are described as applied to a NN filter that enhances or improves the quality of one of its inputs. One example is a NN-based in-loop filter that enhances the quality of an input picture, block or patch in terms of one or more quality metrics, such as the Peak Signal-to-Noise Ratio (PSNR) metric. However, the apparatus, method and computer program product may be valid or applicable to other NNs than a NN filter.

[0092] Additionally, the apparatus, method and computer program product of an example embodiment may be applied to a NN that is used for decoding a data item. However, other embodiments of the apparatus, method and computer program product may be valid for or applicable to a NN that is used for post-processing a decoded data item or data derived from a decoded data item.

[0093] While at least some embodiments are described in relation to input and output data in the form of images or (video) frames or pictures, the apparatus, method and computer program product of other embodiments may be applicable also to other types of data, such as audio frames. Furthermore, while at least some embodiments are described by considering a full image, other embodiments of the apparatus, method and computer program product may be applicable also to one or more blocks or portions of an image.

[0094] It is to be noted that in some cases an encoder includes a decoder or a subset of components of a decoder. For example, a NN-based in-loop filter may be present in both the encoder and the decoder of a codec. In at least some embodiments, a processing unit or processing block may be part of a NN that may be used within an encoder and / or a decoder such as a NN-based in-loop filter or a NN-based post-processing filter. For example, an individual processing unit of an embodiment may be used within an example architecture of an NN in-loop filter.

[0095] As described above, the apparatus of an example embodiment may be embodied by an encoder or decoder, but the apparatus may alternatively be embodied by any of a variety of other computing devices. Regardless of the device that embodies the apparatus, the apparatus 310 of an example embodiment depicted in FIG. 3 may include, for example, at least one processor 312 and at least one memory 314 storing instructions 315 that, when executed by the at least one processor, cause the apparatus 310 at least to perform the method(s) as disclosed herein. In an example, the at least one memory 314 and the instructions 315 (e.g. a computer program code, software), are configured, with the at least one processor 312, to cause the apparatus 310 to perform the method(s) as disclosed herein, and any of the embodiments thereof. For example, the apparatus 310 may be caused or configured to perform at least the method of FIG. 4 and / or any one or more of the embodiments described.

[0096] A processor 312 may comprise circuitry, or be constituted as circuitry or circuitries, the circuitry or circuitries being configured to perform phases of methods in accordance with example embodiments described herein. As used in this application, the term “circuitry” may refer to one or more or all of the following: (a) hardware-only circuitimplementations, such as implementations in only analog and / or digital circuitry, and (b) combinations of hardware circuits and software, such as, as applicable: (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a user equipment, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation. This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0097] The memory 314 may be implemented using any suitable data storage technology. The memory may comprise a database for storing data. The memory 314 may be at least in part external to apparatus 310 but accessible to apparatus 310. The instructions 315 may be comprised in a computer readable medium or a non-transitory computer readable medium. A term non-transitory, as used herein, is a limitation of the medium itself (e.g., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., random access memory (RAM) vs. read only memory (ROM)).

[0098] The apparatus 310 of an example embodiment may include a radio interface 316. The radio interface 316 may provide the apparatus 310 with communication capabilities. The radio interface 316 may comprise a receiver configured to receive information in accordance with at least one cellular or non-cellular standard. The radio interface 316 may comprise a transmitter configured to transmit information in accordance with at least one cellular or non-cellular standard. The receiver may comprise more than one receiver. The transmitter may comprise more than one transmitter. The radio interface 316 may comprise a transceiver configured to receive and transmit information in accordance with at least one cellular or non-cellular standard. The transceiver may comprise more than one transceiver.

[0099] The apparatus 310 may optionally comprise a user interface 318 comprising, for example, at least one of a keypad, a microphone, a touch display, a display, a speaker, etc. The user interface 318 may be used to control the apparatus by the user. The user interface318 may be external to the apparatus 310. For example, the apparatus 310 may be connected to another device, such as a computer, either via wireless or wired connection, and the apparatus 310 is controlled by the user via the computer.

[0100] The operations performed by the apparatus 310 in accordance with an example embodiment are shown in Figure 4. In this regard, the apparatus 310 includes means, such as the at least one processor 312 or the like, for processing an input or data derivative therefrom to generate one or more attention outputs and means, such as the at least one processor 312 or the like, for combining at least the input or data derivative therefrom and the one or more attention outputs to obtain an output. See blocks 402 and 404 of Figure 4.

[0101] In this example embodiment, the instructions, when executed by the at least one processor 312, generate a multi-branch attention block that includes one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output, and wherein the input or data derivative therefrom is input to the multi-branch attention block.

[0102] In an embodiment, a NN filter may include one or more multi-branch attention (MBA) blocks or modules, an MBA block may include at least one or more attention branches and an attention usage block, an input to the MBA block, or data derived from an input to the MBA block, may be processed by the one or more attention branches to obtain a respective one or more attention outputs, and the one or more attention outputs and the input to the MBA block or data derived from the input to the MBA block may be combined by means of an attention usage block to obtain an output of the MBA block. FIG. 5 illustrates an example of this embodiment. By way of example, the MBA block 500 of FIG. 5 may be implemented by the apparatus 310 of FIG. 3.

[0103] As illustrated in FIG. 5, the MBA block 500 includes two attention branches 502 and 504 designated “Attention branch 1” and “Attention branch 2”. The two attention branches 502 and 504 take the input (“Input”) to the MBA block 500 and output the respective two attention outputs 506 and 508 designated as “Attention output 1” and “Attention output 2”. The two attention outputs 506 and 508 and the input to the MBA block 500 are combined by means of the attention usage block 510 designated as “Attention usage” to obtain the output (“Output”) of the MBA block 500.

[0104] In an embodiment, at least one of the one or more attention branches may include an aggregation operation that, for example, aggregates its input to obtain an aggregation output. If an attention branch includes only an aggregation operation, an output of theaggregation operation (e.g., an aggregation output) may represent an output of the attention branch (e.g., an attention output). An example of this embodiment is illustrated in FIG. 6. By way of example, the MBA block 600 of FIG. 6 may be implemented by the apparatus 310 of FIG. 3. An aggregation operation (that may be also referred to as a summarization operation, a reduction operation, or a pooling operation) may be an operation that aggregates (e.g., summarizes, reduces or pools) some aspects or portions or dimensions or sizes of an input to obtain an aggregated (e.g., summarized, reduced or pooled) output. In one example, a size of a tensor that is input to an aggregation operation is reduced. In another example, one or more sizes of respective one or more dimensions of a tensor are reduced. In yet another example, one or more sizes along respective one or more dimensions of a tensor are reduced. In yet another example, a height and a width of a tensor are reduced.

[0105] The MBA block 600 of FIG. 6 includes the first attention branch and the second attention branch, where the first attention branch includes the first aggregation operation 602 designated as “Aggregate 1” and the second attention branch includes the second aggregation block 604 designated as “Aggregate 2”. The first attention branch and the second attention branch take the input (“Input”) to the MBA block 600 and output the first attention output 606 designated as “Attention output 1” and the second attention output 608 designated as “Attention output 2” respectively. The first attention output 606, the second attention output 608, and the input to the MBA block 600 are combined by means of the attention usage block 610 designated as “Attention usage” to obtain the output (“Output”) of the MBA block 600.

[0106] In various embodiments, an input to an MBA block may be a tensor and may be referred to as input tensor, an output of an MBA block may be a tensor and may be referred to as output tensor, intermediate outputs such as an aggregation output, a summarized output, or an attention output may be tensors and may be referred to as an aggregated tensor, a summarized tensor, or an attention tensor respectively. For example, an input to an MBA block may be a 3 -dimensional tensor of shape CxHxW (which may be denoted also as (C, H, W)), where C stands for number of channels, H stands for height, and W stands for width. For example, C may comprise a size of a channel dimension or axis of the tensor, H may comprise a size of a vertical dimension or axis of the tensor, and W may comprise a size of a horizontal dimension of axis of the tensor. In other examples, various tensors may include a different number of dimensions and / or different shapes, such as, for example, a 4-dimensional tensor of shape BxCxHxW (which may be denoted also as (B, C, H, W)), where B stands for batch size (e.g., representing a number of samples in a batch, such as number ofpictures in a batch or number of pictures in a tensor) and other dimensions are similar as for the 3-dimensional example.

[0107] In some embodiments, an aggregation operation may reduce the sizes of one or more dimensions of an input tensor, reduce the height and width of the input tensor to a smaller height and width, reduce the number of channels of the input tensor to a smaller number of channels, comprise a mean or average operation, comprise a sum operation, comprise a median operation, comprise a mode operation (e.g., computing the most frequent value), comprise a minimum operation (e.g., computing the minimum value), comprise a maximum operation (e.g., computing the maximum value), and / or the like.

[0108] In one example, the input tensor has shape CxHxW, the aggregation operation is a mean or average operation computed over the elements of the height and width dimensions (sometimes also referred to as vertical and horizontal dimensions) separately for each channel, and the aggregated tensor has shape Cxlxl, (e.g., the aggregated tensor may have height equal to 1 and width equal to 1).

[0109] In another example, the input tensor has shape CxHxW, the aggregation operation is a mean computed over all channels separately for each spatial element on the height and width dimensions, and the aggregated tensor has shape IxHxW.

[0110] In another example, the input tensor has shape CxHxW, the aggregation operation is a mean computed patch-wise separately over the vertical dimension and the horizontal dimension (e.g., by using patches of size 2x2), and the summarized tensor has shape Ixhxw where h is a height smaller than H and w is a width smaller than W (e.g., h is half of H and w is half of W). In yet another example, the aggregation operation is an average pooling operation. In yet another example, the aggregation operation is a max pooling operation. In yet another example, the aggregation operation is a global average pooling operation.

[0111] In some examples, an input to an MBA block or data derivative therefrom (e.g., data derived based on a first derivation operation) may be an input to an attention branch included within the MBA block. In various examples, an input to an attention branch or data derivative therefrom (e.g., data derivative of a second derivation operation) may be an input to an aggregation operation included within the attention branch. In this manner, in various examples, one or more of the input to an MBA block, the input to an attention branch, and / or the input to an aggregation operation may be the same or may be different.

[0112] FIG. 7 illustrates an example of some of the embodiments described herein. By way of example, the MBA block 700 of FIG. 7 may be implemented by the apparatus 310 of FIG. 3. The MBA block 700 includes the first attention branch and the second attentionbranch, where the first attention branch includes the first aggregation operation 702 designated as “Mean 1” that computes a mean over one or more first dimensions of the input (“Input”) to the MBA block 700 and the second attention branch includes the second aggregation 704 operation designated as “Mean 2” that computes a mean over one or more second dimensions of the input to the MBA block 700. The first attention branch and the second attention branch take the input to the MBA block 700 and output the first attention output 706 designated “Attention output 1” and the second attention output 708 designated “Attention output 2”, respectively. The first attention output 706, the second attention output 708 and the input to the MBA block 700 are combined by means of the attention usage block 710 designated as “Attention usage” to obtain the output (“Output”) of the MBA block 700.

[0113] In some embodiments, at least one of the one or more attention branches may include one or more neural network layers and / or other processing operations. An output of the one or more neural network layers and / or other processing operations may represent an output of the at least one of the one or more attention branches (e.g., an attention output).

[0114] In an embodiment, at least one of the one or more attention branches may include an aggregation operation and one or more neural network layers and / or other processing operations. In various examples, an aggregation operation may be configured to generate an aggregation output that represents an aggregated or reduced or summarized version of an input to the aggregation operation. In certain examples, an input to an aggregation operation may be an input to an attention branch that the aggregation operation is included within. In some examples, an output of the aggregation operation may be an input to the one or more neural network layers and / or other processing operations. An output of the one or more neural network layers and / or other processing operations may represent an output of the at least one of the one or more attention branches (e.g., an attention output). An example of this embodiment is illustrated in FIG. 8. By way of example, the MBA block 800 of FIG. 8 may be implemented by the apparatus 310 of FIG. 3.

[0115] As illustrated in FIG. 8, the MBA block 800 includes the first attention branch and the second attention branch, where the first attention branch includes the first aggregation operation 802 designated as “Aggregate 1” followed by one or more first neural network layers 806 designated as “NN layers 1” and the second attention branch includes the second aggregation operation 804 designated as “Aggregate 2” followed by one or more second neural network layers 808 designated as “NN layers 2”. The first attention branch and the second attention branch take the input (“Input”) to the MBA block 800 and output the first attention output 810 designated as “Attention output 1” and the second attention output 812designated as “Attention output 2”, respectively. The first attention output 810, the second attention output 812 and the input to the MBA block 800 are combined by means of the attention usage block 814 designated as “Attention usage” to obtain the output (“Output”) of the MBA block 800.

[0116] FIG. 9 illustrates a further example of this embodiment. By ways of example, the MBA block 900 of FIG. 9 may be implemented by the apparatus 310 of FIG. 3. The MBA block 900 includes the first attention branch and the second attention branch, where the first attention branch includes the first aggregation operation 902 designated as “Mean 1” that computes a mean over one or more first dimensions of the input (“Input”) to the MBA block 900, followed by the first convolutional layer 906 designated as “Conv layer 1”, and the second attention branch includes the second aggregation operation 904 designated as “Mean 2” that computes a mean over one or more second dimensions of the input to the MBA block 900, followed by the second convolutional layer 908 designated as “Conv layer 2”. The first attention branch and the second attention branch take the input to the MBA block 900 and output the first attention output 910 designated as “Attention output 1” and the second attention output 912 designated as “Attention output 2”, respectively. The first attention output 910, the second attention output 912 and the input to the MBA block 900 are combined by means of the attention usage block 914 designated as “Attention usage” to obtain the output (“Output”) of the MBA block 900.

[0117] In an embodiment, at least one of the one or more attention outputs may be output by the MBA block (e.g., at least one of the one or more attention outputs may be included in an output of the MBA block). FIG. 10 illustrates an example of this embodiment where the second attention output 1012 designated as “Attention output 2” is an output of the MBA block 1000.

[0118] In an embodiment, an input to a second MBA block may include at least one of one or more attention outputs of a first MBA block, where the at least one of one or more attention outputs of the first MBA block may be used within a respective attention branch of the second MBA block as a summarization output, an aggregation output, or the like. For example, the respective attention branch of the second MBA block may not include a summarization operation, an aggregation operation, or the like, and may use the at least one of one or more attention outputs of a first MBA block as an input to one or more neural network layers and / or other processing operations or as an output of the respective attention branch of the second MBA block. In some examples, an MBA block may not include a skipped summarization operation, aggregation operation, or the like. FIG. 11 illustrates anexample of this embodiment. By way of example, the MBA block 1100 of FIG. 11 may be implemented by the apparatus 310 of FIG. 3.

[0119] FIG. 11 illustrates the first MBA block 1100-1 designated as “MBA block 1” and the second MBA block 1100-2 designated as “MBA block 2”. The first MBA block 1100-1 outputs the first output (“Output 1”) and the second output 1102 designated as “Attention output 2”. The second output 1102 is an attention output (e.g., an output of one of the attention branches of the first MBA block 1100-1). The second MBA block 1100-2 is provided with a first input that is the first output of the first MBA block 1100-1 and with a second input that is the second output 1102 of the first MBA block 1100-1. The first input is processed by the third attention branch, including the third aggregation operation 1104 designated as “Aggregate 3” and one or more third neural network layers 1108 designated as “NN layers 3” to obtain the third attention output 1112 designated as “Attention output 3”. The second input (that is the second output 1102 of the first MBA block 1100-1) is processed by the fourth attention branch including one or more fourth neural network layers 1106 designated as “NN layers 4” to obtain the fourth attention output 1110 designated as “Attention output 4”. The third attention output 1112, the fourth attention output 1110 and the first input of the second MBA block 1100-2 are provided as input to the second attention usage block 1114 designated as “Attention usage 2”.

[0120] In some embodiments, the attention usage block may include one or more modulation operations. For example, a modulation operation modulates or modifies a first signal based on a second signal to obtain a modulation output, where the first signal may include an input to the MBA block, and / or data derived from (or obtained based on) an input to the MBA block, and the second signal may include at least one of the one or more attention outputs (e.g., the one or more outputs of respective one or more attention branches of the MBA block).

[0121] In an embodiment, at least two of one or more modulation outputs that are output from respective at least two of the one or more modulation operations, and / or an input to the MBA block and / or data that is derived from (or obtained based on) an input to the MBA block, may be combined by means of a combination operation to obtain a combined output. The combined output may be included in an output of the MBA block. FIG. 12 illustrates an example of this embodiment. By way of example, the MBA block 1200 of FIG. 12 may be implemented by the apparatus 310 of FIG. 3.

[0122] As illustrated in FIG. 12, an MBA block 1200 includes the first attention branch 1202 designated as “Attention branch 1” and the second attention branch 1204 designated as“Attention branch 2”. The first attention branch 1202 takes the input (“Input”) to the MBA block 1200 and outputs the first attention output 1206 designated as “Attention output 1” and the second attention branch 1204 takes the input to the MBA block 1200 and outputs the second attention output 1208 designated as “Attention output 2”. The first attention output 1206, the second attention output 1208 and the input to the MBA block are provided as input to the attention usage block 1210. The attention usage block 1210 includes the first modulation operation 1212 designated as “Modulate 1”, the second modulation operation 1214 designated as “Modulate 2”, and the combination operation 1220 designated as “Combine”. The first modulation operation 1212 takes as input the first attention output 1206 and the input to the MBA block 1200, and outputs the first modulation output 1216 designated as “Modulation output 1”. The second modulation operation 1214 takes as input the second attention output 1208 and the input to the MBA block 1200, and outputs the second modulation output 1218 designated as “Modulation output 2”. The first and second modulation outputs 1216 and 1218 are combined by means of the combination operation 1220 to obtain the output (“Output”) of the MBA block 1200.

[0123] In various embodiments, a modulation operation may comprise one or more of an element-wise multiplication, an element-wise summation, a concatenation operation, one or more neural network layers or the like. In some embodiments, at least two of one or more modulation operations may be the same. For example, two modulation operations may both be multiplication operations. In certain embodiments, at least two of one or more modulation operations may be different. For example, a fist modulation operation may be a multiplication operation and a second modulation operation may be a summation operation. In various embodiments, an input to a modulation operation may include or be a tensor, an output of a modulation operation may be or include a tensor, an input to a combination operation may be or include a tensor, and / or an output of a combination operation may be or include a tensor.

[0124] In an embodiment, when two or more inputs to a modulation operation or a combination operation include respective two or more tensors with different shape, a broadcasting operation in one or more dimensions or axes of the two or more tensors may be performed as part of the modulation operation or as part of the combination operation. In one example, a first input to a multiplication operation is an attention output that includes a first tensor with shape Cxlxl; a second input to the multiplication operation is an input to a MBA block that includes a second tensor with shape CxHxW; the first tensor is modified by broadcasting (or copying or repeating) a value of the first tensor at the i-th index of thechannel dimension to all spatial positions in the other two dimensions to obtain a new version of the first tensor that has shape CxHxW.

[0125] FIG. 13 illustrates an example of some of the embodiments described herein. By way of example, the MBA block 1300 of FIG. 13 may be implemented by the apparatus 310 of FIG. 3. The MBA block 1300 includes the first attention branch 1302 designated as “Attention branch 1” and the second attention branch 1304 designated as “Attention branch 2”. The first attention branch 1302 takes the input (“Input”) to the MBA block 1300 and outputs the first attention output 1306 designated as “Attention output 1” and the second attention branch 1304 takes the input to the MBA block 1300 and outputs the second attention output 1308 designated as “Attention output 2”. The first attention output 1306, the second attention output 1308 and the input to the MBA block 1300 are provided as input to the attention usage block 1310 that includes the first modulation operation 1312 designated as “Multiply”, the second modulation operation 1314 designated as “Sum”, and the combination operation 1320 designated as “NN layers”. The first modulation operation 1312 performs an element-wise multiplication of the first attention output 1306 and the input to the MBA block 1300, and outputs the first modulation output 1316 designated as “Modulation output 1”. The second modulation operation 1314 performs an element- wise summation of the second attention output 1308 and the input to the MBA block 1300, and outputs a second modulation output 1318 designated as “Modulation output 2”. The combination operation 1320 includes one or more neural network layers and / or other operations (e.g., one or more convolutional layers). The first and second modulation outputs 1316 and 1318 are processed by the one or more neural network layers 1320 and / or other operations, to obtain an output of the MBA block 1300. In one example, the first and second modulation outputs 1316 and 1318 are concatenated to form a single data item, such as a single tensor, and the single data item or single tensor is input to the one or more neural network layers 1320 and / or other operations.

[0126] FIG. 14 illustrates another example of some of the embodiments described herein. By way of example, the MBA block 1400 of FIG. 14 may be implemented by the apparatus 310 of FIG. 3. The MBA block 1400 includes a first attention branch 1402 designated as “Attention branch 1” and the second attention branch 1404 designated as “Attention branch 2”. The first attention branch 1402 takes the input (“Input”) to the MBA block 1400 and outputs the first attention output 1406 “Attention output 1” and the second attention branch 1404 takes the input to the MBA block 1400 and outputs the second attention output 1408 designated as “Attention output 2”. The first attention output 1406, the second attention output 1408 and the input to the MBA block 1400 are provided as input to the attention usageblock 1410 that includes the first modulation operation 1412 designated as “Modulate 1” and the second modulation operation 1414 designated as “Modulate 2”. The first modulation operation 1412 takes as input the first attention output 1406 and the input to the MBA block 1400, and outputs the first modulation output 1416 designated as “Modulation output 1”. The second modulation operation 1414 takes as input the second attention output 1408 and the first modulation output 1416, and outputs the second modulation output (“Output”) that may be included in the output of the MBA block 1400.

[0127] FIG. 15 illustrates a further example of this embodiment. By way of example, the MBA block 1500 of FIG. 15 may be implemented by the apparatus 310 of FIG. 3. The MBA block 1500 includes the first attention branch 1502 designated as “Attention branch 1” and the second attention branch 1504 designated as “Attention branch 2”. The first attention branch 1502 takes the input (“Input”) to the MBA block 1500 and outputs the first attention output 1506 designated as “Attention output 1” and the second attention branch 1504 takes the input to the MBA block 1500 and outputs the second attention output 1508 designated as “Attention output 2”. The first attention output 1506, the second attention output 1508 and the input to the MBA block 1500 are provided as input to the attention usage block 1510 that includes the first modulation operation 1512 designated as “Multiply” and the second modulation operation 1514 designated as “Sum”. The first modulation operation 1512 performs an element-wise multiplication of the first attention output 1506 and the input to the MBA block 1500, and outputs the first modulation output 1516 designated as “Modulation output 1”. The second modulation operation 1514 performs an element-wise summation of the second attention output 1508 and the first modulation output 1516, and outputs the second modulation output (“Output”) that may be included in the output of the MBA block 1500.

[0128] FIG. 16 illustrates an example of some of the embodiments described herein. By way of example, the MBA block 1600 of FIG. 16 may be implemented by the apparatus 310 of FIG. 3. The MBA block 1600 includes the first attention branch 1602 designated as “Attention branch 1” and the second attention branch 1604 designated as “Attention branch 2”. The first attention branch 1602 takes the input (“Input”) to the MBA block 1600 and outputs the first attention output 1606 designated as “Attention output 1” and the second attention branch 1604 takes the input to the MBA block 1600 and outputs the second attention output 1608 designated as “Attention output 2”. The first attention output 1606, the second attention output 1608 and the input to the MBA block 1600 are provided as input to the attention usage block 1610 that includes the first modulation operation 1612 designated as “Modulate 1” and the second modulation operation 1614 designated as “Modulate 2”. Thefirst modulation operation 1612 takes as input the first attention output 1606 and the input to the MBA block 1600, and outputs the first modulation output 1616 designated as “Modulation output 1”. The second modulation operation 1614 takes as input the second attention output 1608, the first modulation output 1616 and the input to the MBA block 1600, and outputs the second modulation output (“Output”) that may be included in the output of the MBA block 1600.

[0129] FIG. 17 illustrates a further example of this embodiment. By way of example, the MBA block 1700 of FIG. 17 may be implemented by the apparatus 310 of FIG. 3. The MBA block 1700 includes the first attention branch 1702 designated as “Attention branch 1” and the second attention branch 1704 designated as “Attention branch 2”. The first attention branch 1702 takes the input (“Input”) to the MBA block 1700 and outputs the first attention output 1706 designated as “Attention output 1” and the second attention branch 1704 takes the input to the MBA block 1700 and outputs the second attention output 1708 designated as “Attention output 2”. The first attention output 1706, the second attention output 1708 and the input to the MBA block 1700 are provided as input to the attention usage block 1710 that includes the first modulation operation 1712 designated as “Multiply” and the second modulation operation 1714 designated as “Sum”. The first modulation operation 1712 performs an element-wise multiplication of the first attention output 1706 and the input to the MBA block 1700, and outputs the first modulation output 1716 designated as “Modulation output 1”. The second modulation operation 1714 performs an element-wise summation of the second attention output 1708, the first modulation output 1716 and the input to the MBA block 1700, and outputs the second modulation output (“Output”) that may be included in the output of the MBA block 1700.

[0130] FIG. 18 illustrates an example of some of the embodiments described herein. By way of example, the MBA block 1800 of FIG. 18 may be implemented by the apparatus 310 of FIG. 3. The MBA block 1800 includes the first attention branch 1802 designated as “Attention branch 1” and the second attention branch 1804 designated as “Attention branch 2”. The first attention branch 1802 takes the input (“Input”) to the MBA block 1800 and outputs the first attention output 1806 designated as “Attention output 1” and the second attention branch 1804 takes the input to the MBA block 1800 and outputs the second attention output 1808 designated as “Attention output 2”. The first attention output 1806, the second attention output 1808 and the input to the MBA block 1800 are provided as input to the attention usage block 1810 that includes the first modulation operation 1812 designated as “Modulate 1”, the second modulation operation 1814 designated as “Modulate 2” and thecombination operation 1820 designated as “Combine”. The first modulation operation 1812 takes as input the first attention output 1806 and the input to the MBA block 1800, and outputs the first modulation output 1816 designated as “Modulation output 1”. The second modulation operation 1814 takes as input the second attention output 1808 and the first modulation output 1816 and outputs the second modulation output 1818 designated as “Modulation output 2”. The second modulation output 1818 and the input to the MBA block 1800 are combined by means of the combination operation 1820 to obtain the output (“Output”) of the MBA block 1800.

[0131] FIG. 19 illustrates a further example of this embodiment. By way of example, the MBA block 1900 of FIG. 19 may be implemented by the apparatus 310 of FIG. 3. The MBA block 1900 includes the first attention branch 1902 designated as “Attention branch 1” and the second attention branch 1904 designated as “Attention branch 2”. The first attention branch 1902 takes the input (“Input”) to the MBA block 1900 and outputs the first attention output 1906 designated as “Attention output 1” and the second attention branch 1904 takes the input to the MBA block 1900 and outputs the second attention output 1908 designated as “Attention output 2”. The first attention output 1906, the second attention output 1908 and the input to the MBA block 1900 are provided as input to the attention usage block 1910 that includes the first modulation operation 1912 designated as “Multiply”, the second modulation operation 1914 designated as “Sum” and the combination operation 1920 designated as “NN layers”. The first modulation operation 1912 performs an element-wise multiplication of the first attention output 1906 and the input to the MBA block 1900, and outputs the first modulation output 1916 designated as “Modulation output 1”. The second modulation operation 1914 performs an element-wise summation of the second attention output 1908 and the first modulation output 1916 and outputs the second modulation output 1918 designated as “Modulation output 2”. The combination operation 1920 includes one or more neural network layers (e.g., one or more convolutional layers) and / or other operations. The combination operation 1920 processes the second modulation output 1918 and the input to the MBA block 1900 to obtain the output (“Output”) of the MBA block 1900. In one example, when the second modulation output 1918 and the input to the MBA block 1900 are two tensors, the two tensors may be first concatenated to form a single tensor and the single tensor is input to one or more neural network layers.

[0132] FIG. 20 illustrates an example of some of the embodiments described herein. By way of example, the MBA block 2000 of FIG. 20 may be implemented by the apparatus 310 of FIG. 3. The MBA block 2000 includes the first attention branch and the second attentionbranch, where the first attention branch includes the first aggregation operation 2002 designated as “Aggregate 1” followed by one or more first neural network layers 2006 designated as “NN layers 1” and the second attention branch includes a second aggregation operation 2004 designated as “Aggregate 2” followed by one or more second neural network layers 2008 designated as “NN layers 2”. The first attention branch and the second attention branch take the input (“Input”) to the MBA block 2000 and output the first attention output 2012 designated as “Attention output 1” and the second attention output 2014 designated as “Attention output 2”, respectively. The first attention output 2012, the second attention output 2014 and the input to the MBA block 2000 are combined by means of the attention usage block 2010 to obtain the output (“Output”) of the MBA block 2000. The attention usage block 2010 includes the first modulation operation 2016 designated as “Modulate 1” and the second modulation operation 2018 designated as “Modulate 2”. The first modulation operation 2016 takes as input the first attention output 2012 and the input to the MBA block 2000, and outputs the first modulation output 2020 designated as “Modulation output 1”. The second modulation operation 2018 takes as input the second attention output 2014, the first modulation output 2020 and the input to the MBA block 2000, and outputs the second modulation output that may be included in an output of the MBA block 2000.

[0133] FIG. 21 illustrates a further example of this embodiment. By way of example, the MBA block 2100 of FIG. 21 may be implemented by the apparatus 310 of FIG. 3. The MBA block 2100 includes the first attention branch and the second attention branch, where the first attention branch includes the first aggregation operation 2102 designated as “Mean 1” that computes a mean over one or more first dimensions of the input (“Input”) to the MBA block 2100, such as separately over the vertical dimension and over the horizontal dimension, followed by the first convolutional layer 2106 designated as “Conv layer 1”, and the second attention branch includes the second aggregation operation 2104 designated as “Mean 2” that computes a mean over one or more second dimensions of the input to the MBA block 2100, such as separately over the vertical dimension and over the horizontal dimension, followed by the second convolutional layer 2108 designated as “Conv layer 2”. The first attention branch and the second attention branch take the input to the MBA block 2100 and output the first attention output 2112 designated as “Attention output 1” and the second attention output 2114 designated as “Attention output 2”, respectively. The first attention output 2112, the second attention output 2114 and the input to the MBA block 2100 are combined by means of the attention usage block 2110 to obtain the output of the MBA block 2100. The attention usage block 2110 includes the first modulation operation 2116 designated as “Multiply” andthe second modulation operation 2118 designated as “Sum”. The first modulation operation 2116 performs an element- wise multiplication of the first attention output 2112 and the input to the MBA block 2100, and outputs the first modulation output 2120 designated as “Modulation output 1”. The second modulation operation 2118 performs an element-wise summation of the second attention output 2114, the first modulation output 2120 and the input to the MBA block 2100, and outputs the second modulation output “Output” that may be included in the output of the MBA block 2100.

[0134] In an embodiment, an input to at least one of the one or more modulation operations, such as the first signal, may include an output of an aggregation operation, where an input to the aggregation operation may include the input to the MBA block.

[0135] In various embodiments, when an input to at least one of the one or more modulation operations includes an output of an aggregation operation and an input to the aggregation information includes an input to the MBA block, an output of the at least one of the one or more modulation operations may be combined with an input to the MBA block by means of another modulation operation or a combination operation, such as, for example, a multiplication or a summation.

[0136] As described herein, the apparatus 310 and, more particularly, the instructions 315, when executed by at least one processor 312, is configured to generate a multi-branch attention block that includes one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output, and wherein the input or data derivative therefrom is input to the multi-branch attention block. As such, the performance of a device, such as an encoder, decoder, and / or neural network filter that embodies the apparatus may be improved.

[0137] FIG. 4 is a flowchart illustrating a method according to an example embodiment. It will be understood that each block or signal and combination of blocks and signals may be implemented by various means, such as hardware, firmware, processor, circuitry, and / or other communication devices associated with execution of software including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, the computer program instructions which embody the procedures described above may be stored by the memory 314 of an apparatus 310 employing an example embodiment and executed by at least one processing circuitry 312. As will be appreciated, any such computer program instructionsmay be loaded onto a computer or other programmable apparatus (for example, hardware) to produce a machine, such that the resulting computer or other programmable apparatus implements the functions specified in the flowchart blocks. These computer program instructions may also be stored in a computer-readable memory that may direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture the execution of which implements the function specified in the flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart blocks.

[0138] Accordingly, blocks of the flowchart support combinations of means for performing the specified functions and combinations of operations for performing the specified functions. It will also be understood that one or more blocks of the flowchart, and combinations of blocks in the flowchart, can be implemented by special purpose hardwarebased computer systems which perform the specified functions, or combinations of special purpose hardware and computer instructions.

[0139] In an embodiment, at least some of the processes described herein may be carried out by an apparatus comprising means for carrying out at least some of the described processes. Means for performing method steps as disclosed herein may include software and / or hardware components of the apparatus 310. For example, the at least one processor 312, the memory 314, and the computer program code form means for carrying out the method or methods as disclosed herein, and any of the embodiments thereof. As used herein the term “means” is to be construed in singular form, e.g., referring to a single element, or in plural form, e.g., referring to a combination of single elements. Therefore, terminology “means for [performing A, B, C]”, is to be interpreted to cover an apparatus in which there is only one means for performing A, B and C, or where there are separate means for performing A, B and C, or partially or fully overlapping means for performing A, B, C. Further, terminology “means for performing A, means for performing B, means for performing C” is to be interpreted to cover an apparatus in which there is only one means for performing A, B and C, or where there are separate means for performing A, B and C, or partially or fully overlapping means for performing A, B, C.

[0140] Even though the present disclosure has been described above with reference to an example according to the accompanying drawings, it is clear that the present disclosure is not restricted thereto but can be modified in several ways within the scope of the appended claims. Therefore, all words and expressions should be interpreted broadly and they are intended to illustrate, not to restrict, the embodiment. It will be obvious to a person skilled in the art that, as technology advances, the inventive concept can be implemented in various ways. Further, it is clear to a person skilled in the art that the described embodiments may, but are not required to, be combined with other embodiments in various ways.

Claims

THAT WHICH IS CLAIMED:

1. An apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to perform at least:process an input or data derivative therefrom to generate one or more attention outputs; andcombine at least the input or data derivative therefrom and the one or more attention outputs to obtain an output,wherein the instructions, when executed by the at least one processor, generate a multi-branch attention block comprising one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output, and wherein the input or data derivative therefrom is input to the multi-branch attention block.

2. The apparatus according to claim 1, wherein the input or data derivative therefrom that is input to the multi-branch attention block is an input tensor, the output is an output tensor, and the one or more attention outputs are attention tensors.

3. The apparatus according to claim 1 or 2, wherein at least one attention branch of the one or more attention branches comprises at least one aggregation operation configured to generate an aggregation output that represents an aggregated version of an input to the at least one aggregation operation, wherein the input to the at least one aggregation operation is the input or data derivative therefrom.

4. The apparatus according to claim 3, wherein the at least one aggregation operation receives an input tensor and is configured to reduce a size of one or more dimensions of the input tensor or reduce a height and a width of the input tensor or reduce a number of channels of the input tensor.

5. The apparatus according to claim 3, wherein the at least one aggregation operation comprises one or more of a mean operation, average operation, summation operation, median operation, mode operation, minimum operation, or maximum operation.

6. The apparatus according to any one of claims 1 to 5, wherein at least one attention branch of the one or more attention branches comprises at least one or more neural network layers.

7. The apparatus according to any one of claims 1 to 6, wherein the output is an output of the multi-branch attention block and comprises at least one attention output of the one or more attention outputs.

8. The apparatus according to claim 7, wherein the instructions, when executed by the at least one processor, further cause the apparatus to generate a second multi-branch attention block that is configured to receive the output comprising the at least one attention output, wherein the second multi-branch attention block comprises at least one attention branch configured to skip at least one aggregation operation by using the at least one attention output as an aggregation output.

9. The apparatus according to any one of claims 1 to 8, wherein the attention usage block comprises at least one or more modulation operations configured to modulate or modify a first signal based on a second signal to obtain a modulation output, wherein the first signal comprises at least the input or data derivative therefrom that is input to the multibranch attention block and the second signal comprises at least one or more of the one or more attention outputs.

10. The apparatus according to claim 9, wherein the instructions, when executed by the at least one processor, further cause the apparatus to combine at least two or more modulation outputs of respective two or more modulation operations and the input or data derivative therefrom via a combination operation to obtain a combined output, wherein the combined output is comprised in the output.

11. The apparatus according to claim 9 or 10, wherein a modulation operation of the at least one or more modulation operations comprises an element wise multiplication or an element wise summation.

12. The apparatus according to any one of claims 9 to 11, wherein at least two modulationoperations are the same.

13. The apparatus according to any one of claims 9 to 11, wherein at least two modulation operations are different.

14. The apparatus according to any one of claims 9 to 13, wherein an input to at least one modulation operation or an output of at least one modulation operation comprises a tensor.

15. The apparatus according to claim 10, wherein an input to the combination operation or an output of the combination operation comprises a tensor.

16. The apparatus according to claim 10, wherein two or more inputs to the respective two or more modulation operations or the combination operation comprise two or more tensors with different shapes, and wherein the instructions, when executed by the at least one processor, further cause the apparatus to perform a broadcasting operation in one or more dimensions or axes of the two or more tensors.

17. The apparatus according to any one of claims 1 to 16, wherein the attention usage block comprises a first modulation operation and a second modulation operation, and wherein the multi-branch attention block comprises at least:a first attention branch comprising at least a first aggregation operation and one or more first neural network layers; anda second attention branch comprising at least a second aggregation operation and one or more second neural network layers;wherein:the first aggregation operation is configured to receive the input or data derivative therefrom and output a first aggregation output,the second aggregation operation is configured to receive the input or data derivative therefrom and output a second aggregation output,the one or more first neural network layers are configured to receive the first aggregation output and output a first attention output,the one or more second neural network layers are configured to receive the second aggregation output and output a second attention output,the first modulation operation is configured to receive the first attention outputand the input or data derivative therefrom and output a first modulation output, and the second modulation operation is configured to receive the first modulation output, the second attention output, and the input or data derivative therefrom and output the output of the multi-branch attention block.

18. The apparatus according to claim 17, wherein the first aggregation operation comprises a first mean operation, the second aggregation operation comprises a second mean operation, the one or more first neural network layers comprise a first convolutional layer, the one or more second neural network layers comprise a second convolutional layer, the first modulation operation comprises a multiplication operation and the second modulation operation comprises a summation operation.

19. A method comprising:processing an input or data derivative therefrom to generate one or more attention outputs; andcombining at least the input or data derivative therefrom and the one or more attention outputs to obtain an output,wherein a multi-branch attention block comprises one or more attention branches configured to process the input or data derivative therefrom to generate the one or more attention outputs, and an attention usage block configured to combine at least the input or data derivative therefrom and the one or more attention outputs to generate the output, and wherein the input or data derivative therefrom is input to the multi-branch attention block.

20. The method according to claim 19, wherein the input or data derivative therefrom that is input to the multi-branch attention block is an input tensor, the output is an output tensor, and the one or more attention outputs are attention tensors.

21. The method according to claim 19 or 20, wherein at least one attention branch of the one or more attention branches comprises at least one aggregation operation configured to generate an aggregation output that represents an aggregated version of an input to the at least one aggregation operation, wherein the input to the at least one aggregation operation is the input or data derivative therefrom.

22. The method according to claim 21, wherein the at least one aggregation operationreceives an input tensor and is configured to reduce a size of one or more dimensions of the input tensor or reduce a height and a width of the input tensor or reduce a number of channels of the input tensor.

23. The method according to claim 21, wherein the at least one aggregation operation comprises one or more of a mean operation, average operation, summation operation, median operation, mode operation, minimum operation, or maximum operation.

24. The method according to any one of claims 19 to 23, wherein at least one attention branch of the one or more attention branches comprises at least one or more neural network layers.

25. The method according to any one of claims 19 to 24, wherein the output is an output of the multi-branch attention block and comprises at least one attention output of the one or more attention outputs.

26. The method according to claim 25, further comprising generating a second multibranch attention block that is configured to receive the output comprising the at least one attention output, wherein the second multi-branch attention block comprises at least one attention branch configured to skip at least one aggregation operation by using the at least one attention output as an aggregation output.

27. The method according to any one of claims 19 to 26, wherein the attention usage block comprises at least one or more modulation operations configured to modulate or modify a first signal based on a second signal to obtain a modulation output, wherein the first signal comprises at least the input or data derivative therefrom that is input to the multibranch attention block and the second signal comprises at least one or more of the one or more attention outputs.

28. The method according to claim 27, further comprising combining at least two or more modulation outputs of respective two or more modulation operations and the input or data derivative therefrom via a combination operation to obtain a combined output, wherein the combined output is comprised in the output.

29. The method according to claim 27 or 28, wherein a modulation operation of the at least one or more modulation operations comprises an element wise multiplication or an element wise summation.

30. The method according to any one of claims 27 to 29, wherein at least two modulation operations are the same.

31. The method according to any one of claims 27 to 29, wherein at least two modulation operations are different.

32. The method according to any one of claims 27 to 31, wherein an input to at least one modulation operation or an output of at least one modulation operation comprises a tensor.

33. The method according to claim 28, wherein an input to the combination operation or an output of the combination operation comprises a tensor.

34. The method according to claim 28, wherein two or more inputs to the respective two or more modulation operations or the combination operation comprise two or more tensors with different shapes, and further comprising performing a broadcasting operation in one or more dimensions or axes of the two or more tensors.

35. The method according to any one of claims 19 to 34, wherein the attention usage block comprises a first modulation operation and a second modulation operation, and wherein the multi-branch attention block comprises at least:a first attention branch comprising at least a first aggregation operation and one or more first neural network layers; anda second attention branch comprising at least a second aggregation operation and one or more second neural network layers;wherein:the first aggregation operation is configured to receive the input or data derivative therefrom and output a first aggregation output,the second aggregation operation is configured to receive the input or data derivative therefrom and output a second aggregation output,the one or more first neural network layers are configured to receive the firstaggregation output and output a first attention output,the one or more second neural network layers are configured to receive the second aggregation output and output a second attention output,the first modulation operation is configured to receive the first attention output and the input or data derivative therefrom and output a first modulation output, and the second modulation operation is configured to receive the first modulation output, the second attention output, and the input or data derivative therefrom and output the output of the multi-branch attention block.

36. The method according to claim 35, wherein the first aggregation operation comprises a first mean operation, the second aggregation operation comprises a second mean operation, the one or more first neural network layers comprise a first convolutional layer, the one or more second neural network layers comprise a second convolutional layer, the first modulation operation comprises a multiplication operation and the second modulation operation comprises a summation operation.

Images