Dynamic slimmable neural network for sequential data processing

Abstract

There is disclosed an apparatus (100, 200, 300, 400, 500) comprising: an input interface (101) to receive an input segment (102) of an input sequential data; a neural network, NN, processor (110) to derive an output result (104) by processing the input segment (102) through a NN having a number of layers (111) from a first layer to a last layer, the NN using a predetermined number of deactivatable units which are selectively deactivatable; a gating module (120) configured to deactivate at least one deactivatable unit based on the input segment (102) and / or on at least one intermediate output segment (112, 123), an output interface (103) configured to provide an output (104) derived from the last layer of the NN.

Description

[0001] Dynamic Slimmable Neural Network for Sequential Data Processing

[0002] Description

[0003] This document proposes a computationally efficient system for processing sequential data, such as audio and video signals. The proposed method comprises a neural network capable of adapt- ing its computational complexity dynamically (e.g., over time) based on the characteristics of the input data points of the sequence. The proposed method extends a slimmable neural network, which can be executed at different widths (e.g., number of units) of its layers, with a gating mechanism that dynamically determines the required computational complexity from the input data points. A block diagram of the proposed method is depicted in Fig. 1 .

[0004] Sequential data refers to any set of data points that are arranged in a specific order, for ex- ample based on time (e.g., audio and video signals) and position (e.g., text). Sequential data pro- cessing involves the use of processing systems (e.g., algorithms) that analyze, modify, or synthesize sequential data. This broad term encompasses several application fields such as natural language processing (e.g., machine translation), audio signal processing (e.g., speech enhancement), and video signal processing (e.g., super resolution).

[0005] Prior-art sequential data processing methods employ deep neural networks (DNNs) due to their exceptional ability in modelling long-term dependencies. However, DNNs are often character- ized by a high computational complexity and a large memory footprint, hindering their deployment in resource-constrained edge devices (e.g., mobile phones). In addition, a considerable source of computational inefficiency of typical DNNs employed for processing sequential data is their static nature, where all data points in the input sequence are processed with a uniform computational cost, including those that might not necessitate intensive computations (e.g., silent time frames in a speech enhancement application). With this report, we propose a method that makes the computational cost of a sequential data processing system a function of the input data points.

[0006] Dynamic Neural Networks: Dynamic neural networks are capable of adapting their com- putational graph during inference in an input-dependent fashion

[0101] . In the following, we provide a list of related works, organized according to the employed technique to achieve dynamic computa- tion.

[0007] Mixture-of-experts (MoEs): A MoE model

[0102] comprises independent sub-networks (i.e., experts) and gating modules that dynamically route the input to such experts. Since the sub- networks in MoEs are independent (i.e., do not share weights), the memory footprints of MoEs in- creases when the number of experts is large. MoEs have been employed in sequential data processing applications, such as language modeling (e.g., Gemini 1.5

[0103] ), acoustic modeling (e.g.,

[0104] ) and speech enhancement (e.g.,

[0105] ). Dynamic depth: Early exiting

[0106] and layer skipping

[0107] are two mechanisms that dy- namically vary the depth (i.e., number of layers) of a DNN at run-time based on the input character- istics, and therefore allow reducing unnecessary computations. Early existing enables a DNN to stop execution at an intermediate layer during inference based on a particular criterion, e.g., by comparing the predictions of two consecutive layers. In contrast, layer skipping utilizes a gating mechanism to determine whether a layer should be executed or not. Layer skipping is typically employed in DNNs with skip connections to ensure the continuity of forward propagation. In the context of speech en- hancement, several works have utilized early existing (e.g., [108, 109]) and layer skipping (e.g.,

[0110] ) to achieve dynamic computation at run-time.

[0008] Dynamic width: Slimmable neural networks (SNNs)

[0111] are a class of networks that can be executed using different widths (e.g., number of units in a feedforward layer or number of kernels in a convolutional layer) of their layers at run-time to save computations. However, SNNs are not dynamic as the width of the slimmable layers is set manually by the user during inference (e.g., based on a computational budget), rather than based on the input characteristics. SNNs have been proposed to allow re-configuring a speech enhancement system at run-time to meet varying resource con- straints

[0112] .

[0009] Dynamic SNNs allow varying the width of their layers based on the input characteristics. In

[0013] , a dynamic SNN operating on images (i.e., non-sequential data) has been proposed, where the utilization of each slimmable layer is predicted by a gate that globally analyzes its input features. However, extending dynamic SNNs to sequential data processing has not yet been explored. The proposed method, described in Section 3, employs a dynamic SNN specifically designed to operate on sequential data. In contrast to

[0113] , where the gating mechanism globally analyzes the input sample (or intermediate features), the proposed method utilizes a context-aware gating mechanism that performs local processing on the input sequence to dynamically adapt the SNN’s width for each segment in the sequence.

[0010] In general terms, neural networks can cause a waste of computational power in the cases in which a result does not require too much complexity, in particular in cases like video and audio processing.

[0011] Summary

[0012] There is provided an apparatus comprising: an input interface to receive an input segment of an input sequential data; a neural network, NN, processor to derive an output result by processing the input segment through a NN having a number of layers from a first layer to a last layer, the NN using a predetermined number of deactivatable units which are selectively deactivatable; a gating module configured to deactivate at least one deactivatable unit based on the input segment and / or on at least one intermediate output segment, an output interface configured to provide an output derived from the last layer of the NN.

[0013] There is proposed a method comprising: receiving an input segment of an input sequential data through a neural network, NN, deriving an output result by processing the input segment through the NN having a number of layers from a first layer to a last layer, the NN having a prede- termined number of deactivatable units; deactivating at least one deactivatable unit based on the input segment and / or on at least one intermediate output segment; outputting an output result derived from the last layer of the NN.

[0014] There is proposed a non-transitory storage unit storing instructions which, when ex- ecuted by a processor, cause the processor to perform the method above.

[0015] Figures

[0016] Fig. 1 shows an implementation according to an example.

[0017] Fig. 2 shows an implementation according to an example (which may be an embodiment of the implementation of Fig. 1).

[0018] Fig. 3 shows an implementation according to an example (which may be an embodiment of the example of Fig. 1).

[0019] Fig. 4A shows an example of operation.

[0020] Fig. 4B shows an example of operation with the deactivation of one neuron.

[0021] Fig. 4C shows an example of operation with a symmetrical deactivation of neurons among the lay- ers.

[0022] Fig. 4D shows an example with asymmetrical deactivations of neurons.

[0023] Fig. 4E shows an example of an operation.

[0024] Fig. 4F shows an example of operation based on an intermediate output.

[0025] Fig. 5 A shows an implementation according to an example which may be an embodiment of any of Figs. 1 -3.

[0026] Fig. 5B shows an implementation according to an example which may be an embodiment of any of the examples of any of Figs. 1 -3 and 5 A.

[0027] Fig. 6A shows an example of an implementation.

[0028] Fig. 6B shows an example of an implementation in which some connections are deactivated. Fig. 7A shows an implementation according to the present examples compared to prior art embod- iments.

[0029] Fig. 7B shows an example of the functioning of a present implementation. Fig. 7C shows an example of the functioning of a present implementation.

[0030] Examples

[0031] Fig. 1 shows an example of an apparatus 100 according to the present examples. The appa- ratus 100 may be, for example, a signal processing apparatus (signal processor apparatus). The ap- paratus 100 may be an apparatus for speech enhancement, but this is not exclusive. For example, the apparatus 100 may be for noise reduction, target speaker extraction or, more in general, target com- ponent extraction. The apparatus 100 may be an apparatus for speech separation. The apparatus 100 may be an apparatus for media enhancement and / or video enhancement. The apparatus 100 may include an input interface 101 to receive an input segment 102 (which is here mathematically indi- cated with It).

[0032] The input segment 102 may be a segment of an input sequential data (e.g. an input sequential signal, such as a signal in a time domain (e.g. having a sequence of time domain samples) or a fre- quency domain (e.g. having a time-based sequence such as a sequence of time frequency bins) or a position domain (such as text, which provides information or speech, e.g. based on the position of a phoneme). The input sequential data 102 may be, for example, an audio signal (e.g. evolving in time) or a video signal (e.g. evolving in time or, e.g. as a sequence of video images). The input sequential data 102 may be, more in general, a media signal (e.g. having audio signals and video signals com- bined with each other). The input sequential data 102 may be in the form of text.

[0033] The apparatus 100 may include a neural network processor 1 10 (NN processor). The NN processor 1 10 may process the input segment 102 to obtain an output result 104 (Ot). The output result 104 may be a processed version of the input segment 102. The output result 104 may provide a classification of the input segment 102 or in any case an inference (e.g. a continuous regression). It will be shown that the neural network (NN) may be a slimmable NN (e.g. it may change its com- plexity by deactivating some deactivatable elements (deactivatable units) based e.g. on the input segment 102 and / or other signals, as it will be shown later, such as 105, 1 12, 123 in feedback, etc.). The slimmable NN may have a number of layers from a first layer to a last layer (e.g. it may have at least one or two layers towards a last layer M). In examples, the number of layers is fixed, but some layers (or all layers) have deactivatable units (e.g. neurons, connections, patterns, etc.). In some cases, at least one layer is deactivatable completely. The NN may use a predetermined number of deacti- vatable units which are selectively deactivatable. Therefore, it is possible to selectively activate and / or deactivate some of the deactivatable units. It is to be noted that, in some examples, the concept o f layers may be generalized in several cases with the concept of groups (e.g. groups of deact i vatable units, such as neurons and / or connections and / or patterns) which do not necessarily constitute a layer, or which constitute more than one layer.

[0034] It is important to note that the deactivatable units may differ from example to example. In general terms, the deactivatable units may be (e.g. in the case of fully-connected NNs) neurons and / or connections between neurons (e.g. each connection having a weight), biases, input elements, learna- ble parameters of the NN, while the deactivatable units may be for example patterns (e.g. in the case of convolutional NNs). Therefore, in the case of a convolutional NN, the number of patterns (or their dimension) may be restricted in some cases.

[0035] In the case of the NN being a fully-connected NN, deactivatable units may be neurons. Also, deactivatable units can be biases (e.g. to be added to all the results). Deactivatable units may be connections between neurons (each connection having a weight to be applied by scaling the output of the immediately preceding neuron). Deactivatable units may also be input elements of the input segment 102 (e.g., when some input elements are deactivated, then the dimensionality of the input segment 102 result restricted). Deactivatable units may be learnable parameters of the NN. In the case of the NN being a convolutional neural network, deactivatable units may be patterns (or ele- ments of the patterns, so that that the deactivation causes the reduction of the dimensionality of the pattern(s)).

[0036] The apparatus 100 may include an output interface 103 providing an output 104 derived from the last layer of the neural network. The output 104 may be subjected to thresholds, for example, to provide a classification output or may be used as a processed version of the input segment 102. The NN processor 1 10 may have the NN to be trained to perform a particular operation onto the input segment 102.

[0037] The apparatus 100 may include a gating module 120 configured to deactivate at least one deactivatable unit. In some examples, the gating module 120 may deactivate a number of deactivat- able unit between 0 deactivatable units and all the deactivatable units, but more in particular between 1 deactivatable unit and a certain number of deactivatable units (e.g. based on a classification result); in some other examples, the gating module 120 may be configured to select between deactivating no deactivatable unit at all and deactivating the at least one deactivatable unit, e.g. based on a classifi- cation result. It is here in theory possible that some units are non-deactivatable: e.g. it may be that some neurons are non-deactivatable, while other neurons are deactivatable.

[0038] The gating module 120 may be, for example, a predictor, that predicts the complexity of the processing to be carried out by the NN. The predictor may therefore infer how many deactivatable units are to be used, e.g. based on processing with low complexity. Internally, the gating module 120 may use a NN as well, but in some cases the gating module 120 may use a deterministic processing. In many examples, however, the gating module 120 may be understood as perftirining a classification on the input segment 102, so as to decide how many (and / or which ones of the) deactivatable units to deacti vate, e.g. for each frame of the input segment 102.

[0039] As can be seen, in some cases, the gating module 120 may make use, for carrying out its classification (or in any case its inference, e.g. regression), of the input segment 102. Alternatively or in addition, the gating module 120 may make use of intermediate features 1 12 or 123 for the slimmable NN (e.g. from intermediate, hidden layers). In some cases, the gating module 120 may make use of both the input segment 102 and of one or more of intermediate features 1 12 or 123 from the hidden layers of the NN. As can be seen in Fig. 1 , the gating module 120 may provide a command (here indicated with 122) for deactivating at least one deactivatable unit of the NN. As can be seen, this command 122 may often (but not uniquely) be embodied by a utilization factor ut.

[0040] For example, the gating module 120 may decide how many deactivatable units are to be deactivated. For example, the gating module 120 may decide the number of deactivatable units to be deactivated may be an absolute number (such as a cardinal number of deactivatable units to be deac- tivated) or a ratio (e.g. operating as a percentage) of deactivatable units to be deactivated out of the total deactivatable units] [the gating module 120 may output a number (either as an absolute number or relative number) of layers (or groups of deactivatable units) to be non-deactivated, and may be expressed as a utilization factor utin the interval between 0 (or 0%) and 1 (or 100%), wherein a low value (e.g. close to 0) implies most of the deactivatable units of the at least one layer (or at least one group of deactivatable units) are to be deactivated, while a value of 1 implies that all the deactivatable units of the at least one layer (or at least one group of deactivatable units) are activated and no deac- tivatable unit is deactivated at all, and a value close to 1 implies that almost all the deactivatable units are activated and only some deactivatable units are deactivated; in some examples, a special rule may be defined, that a utilization factor utequal to 0 for one particular layer may imply that the specific layer (or at least one group of deactivatable units) is completely skipped; in some examples, a selec- tion between a first number of deactivations and a second number of deactivations may be carried out, where the first number of deactivations (e.g. 1 or more than 1) is greater than the second number of deactivations (0 or more than 0, but less than the first number of deactivations)] [e.g. in case multiple layers (or groups of deactivatable units) are subjected to deactivation, in some examples for each layer (or group of deactivatable units) subjected to deactivation a layer-specific number of de- activatable units to be deactivat ed (or group-specifi c number of deactivatable units to be deactivated) may be decided by the gating module, and in alternative examples multiple layers or groups of deac- tivatable units (e.g. all the layers or groups of deactivatable units) may be subjected to the same number of deactivatable units to be deactivated (or, more in particular, to the same relative number (e.g. percentage) of deactivatable units to be deactivated)]. In the cases in which the gating module 120 decides that a deactivation is necessary, it may also decide whether and / or how many deactivatable units to deactivate deterministically. For exam- ple, the first deactivatable units to be deactivated may be based on a pre-defined scheme.

[0041] In other cases, the gating module 120 may decide whether and / or how many deactivatable units to deactivate randomly [e.g. based on a random deactivation in such a way that the deactivated deactivatable units are selected randomly] [e.g. in some cases, however, the random decision only interests some types of deactivatable units to be deactivated; for example, in the case in which the deactivatable units are both neurons and connections (e.g. each connection being associated with a particular weight), then the random deactivation may only interest one particular type (e.g. only neu- rons but no connections, or only connections but no neuron, or both neurons and connections)].

[0042] The gating module 120 may decide which deactivatable units to deactivate, e.g. the gating module 120 may be configured to deactivate the at least one deactivatable unit based on the input segment 102 or based on the intermediate output segment 1 12 or 123 outputted by the layer (or group of deactivatable units) preceding (e.g. immediately preceding) the at least one layer or group of de- activatable units. (Instead of the preceding layer, it could be possible to combine the outputs of some of the preceding layers, e.g. via skip connections.)

[0043] In some cases, the gating module 120 may decide how many and / or which deactivatable units to deactivate deterministically based on a ranking of the deactivatable units. For example, more important deactivatable units may have a higher ranking, and may be deactivated last, while less important deactivatable units may be deactivated with a higher probability than the higher ranking deactivatable units. It will be shown that the order of deactivation may follow the particular classifi- cation result obtained by the gating unit 120.

[0044] In some examples, the gating unit 120 may decide which deactivatable units to deactivate based on the weight values of the connections entering in, and / or exiting out, each deactivatable unit, so as to provide a lower ranking to the deactivatable units that have comparatively low weight values of the connections entering and / or exiting [e.g., in average, e.g. in norm, such as magnitude] than deactivatable units that have comparatively high weight values of the connections entering and / or exiting (e.g. in average, e.g. in norm, such as magnitude) [e.g. in case of deacti vation of some deac- tivatable units (e.g. neurons, patterns etc.) in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those that have a low value (e.g. in norm, such as magnitude) because their scaling would imply a computational cost with low benefits; this is in particular im- portant in the cases in which the deactivation is based on the input segment; e.g. in case of low noisiness it may be preferable to deactivate the deactivatable units which have entering and / or exiting connections having weight values (e.g. in average, e.g. in norm, such as magnitude) 0 or closer to 0, to reduce the computational complexity, while in case of higher noisiness it may be preferred to notwithstanding maintain also those deactivatable units, to increase the model capacity]. In addition or alternatively, it is possible for the gating module 120 to decide which deacti- vatable units to deactivate based on their importance quantified by a score function of at least the weight values of the connections entering in and / or exiting out the deactivatable units being neurons [and / or based on a score function computed from the weights values (e.g., norm, correlation) and / or the inputs (e.g. to include activation)] [e.g. in case of deactivation of some deactivatable units in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those that have a low importance because their scaling would imply a computational cost with low benefits; this is particularly important in the cases in which the deactivation is based on the input segment; e.g. in case of non-important deactivatable unit, it may be preferable to avoid it, while in case of increased importance of the deactivatable unit, it may be preferable to notwithstanding maintain activated also those deactivatable units, to increase the quality].

[0045] In addition or alternatively (and how it will be shown for example in Fig. 3), the gating module 120 may decide which deactivatable units [e.g. neurons or patterns] to deactivate based on the weighted values inputted to the same deactivatable units or outputted by the same deactivatable units, so as to provide a lower ranking to the deactivatable units that have comparatively low weighted inputted values or low weighted outputted values [e.g. in average, e.g. in norm, such as magnitude] than deactivatable units that have comparatively high weight values entering have com- paratively high weighted inputted values or high weighted outputted values [e.g. in average, e.g. in norm, such as magnitude] [e.g. in case of deactivation of some deactivatable units (e.g. neurons, pat- terns etc.) in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those deactivatable units that output (or that receive in input) a low value (e.g. in norm, such as magnitude) because their scaling would imply a computational cost with low benefits; this is partic- ularly important in the cases in which the deactivation is based on the input segment. For example Fig. 3 shows that the output 1 12 (123) of each layer (which is input of the immediately subsequent layer) can be provided to a gating unit 121 of the gating module 120, and the resulting command 122 (125) from the gating unit 121 is therefore based on the output 1 12 (123) of that layer (intermediate result).

[0046] In addition or alternatively, the gating module 120 may decide which deactivatable units to deactivate, the deactivatable units being connections between neurons and having weights, so as to deactivate the deactivatable units based on the weight values of the deactivatable units being con- nections, so as to provide a lower ranking to the deactivatable units being connections that have comparatively low weight values [e.g. in average, e.g. in norm, such as magnitude] than deactivatable units being connections that have comparatively high weight values (e.g. in average, e.g. in norm, such as magnitude) [e.g. in case of deactivation of some deactivatable units being connections in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those that have a low weight value (e.g. in norm, such as magnitude) because their scaling would imply a computa- tional cost with low benefits; e.g. in case of low noisiness it may be preferable to deactivate the deactivatable units being connection having weight values (e.g., in average, e.g. in norm, such as magnitude) 0 or closer to 0 than those having higher value, to reduce the computational complexity, while in case of higher noisiness it may be preferred to notwithstanding maintain also those deacti- vatable units (e.g. lower ranked deactivatable units), to increase the model capacity].

[0047] As explained above, the command 122 may be expressed as a utilization factor ufso that the deactivatable units of are ranked from a first-ranked (e.g. highest-ranked) deactivatable unit to a last- ranked (e.g. lowest-ranked) deactivatable unit (in some cases with interposition of at least one inter- mediately-ranked deactivatable unit, e.g. a plurality of intermediately-ranked deactivatable units).A comparatively high utilization factor implies a comparatively high number of activated deactivatable units (including lower ranked deactivatable units), and a comparatively low utilization factor implies a comparatively low number of activated deactivatable units (e.g. no deactivatable unit is deactivated at all or only the lowest ranked ones are deactivated). The deactivatable units are activated in accord- ance to their decreasing ranking (from the first being the highest to the last being the lowest), at the increase of the utilization factor [the least (lowest) ranked deactivatable unit(s) may therefore be the least important, and the highest (first) ranked deactivatable unit unit(s) may therefore be the most important] [the lower the ranking of a deactivatable unit, the higher the utilization factor shall be to cause its activation, and the higher the ranking of a deactivatable unit, the lower the utilization factor can be to cause its activation] [e.g. in the case, like in Fig. 3, of multiple gating units 121 associated with different layers (or groups of deactivatable layers), each gating unit 121 may be configured to apply a utilization factor independently of the other gating units 121],

[0048] In general terms, the ranking of the different deactivatable units may be predefined. For ex- ample, the ranking may be predefined before training. Therefore, it may be that, when the training starts, the ordering defined by the ranking is already defined. In other cases, the ranking may be defined by training.

[0049] In some cases, the ranking of the deactivatable units (in particular, when the deactivatable units are neurons) is conditioned (e.g. at least partially) by the weight values of connections entering in and / or exiting out the deactivatable units (or at least a group or at least one deactivatable unit). For example, the higher (e.g. in module, in norm, in integral, in average, etc.) the weight values of the connections entering in and / or exiting out the deactivatable unit, the higher the ranking. E.g. if a first deactivatable unit has higher (e.g. in module, in norm, in integral, in average, etc.) weight values than a second deactivatable unit, then the first deactivatable unit is given higher ranking than the second deactivatable unit.

[0050] In examples, the ranking may be derived by reordering the deactivatable units based on a score function of the weight values of the connections entering in and / or exiting out the deactivatable units, (this in particular when the deactivatable units are neurons). The score function may be or involve, for example, the magnitude of the weight values (e.g. the highest the weight value in norm, the highest its ranking). In of multiple weight function entering and / or exiting the deactivatable unit, it is possible to derive the ranking based on reordering the deactivatable units based on a score func- tion of aggregated metrics (e.g. integral values, average values, etc.) of the weight values of the connections entering in and / or exiting out the deactivatable units

[0051] In some cases (like in Figs. 2, 5A, 5B) the gating module 120 may apply one single utilization factor to all (or at least a plurality of) deactivatable units of the NN. In other examples, (e.g., but not strictly necessarily, when the NN is a fully-connected NN) it may be possible to provide one specific utilization factor for each layer (e.g. l / below). Therefore, there can be a layer-specific utilization factor for each layer, and the layer-specific utilization factor may vary for each layer (or more in general, for each group of deacti vatable units).

[0052] It is therefore possible that at least one group of deactivatable units (e.g. layers in a fully- connected NN) have the same utilization factor, while other layers (or more in general, other groups of deactivatable units) have different utilization factors. In some cases, different groups of deactivat- able units have no deactivatable unit in the same layer.

[0053] Therefore, there may be a layer-specific utilization factor for each layer (in particular, in fully-connected NNs). [Instead of a layer-specific utilization factor for each layer, it is possible to use a group-specific utilization factor for each group of layers].

[0054] Fig. 2 shows an example of an apparatus 200 (which may be a particular embodiment of the apparatus 100 of Fig. 1 , in which the utilization factor utis provided through the command 122 to some of or all the layers of the NN processed by the NN processor 1 10).

[0055] As can be seen from Fig. 2, all the layers have in this case the same utilization factor. For this reason, they have the same percentage of activated units and all the same percentage of deac- tivated units. The quantity of activate units is indicated with a bar internal to the block that shows the layer 1 1 1 : a utilization factor 1 (or 100%) will imply the complete superposition of the bar to the block indicating the layer 1 1 1 ; and a utilization factor 0 (0%) will imply the complete absence of the bar in the block indicating the layer 111 ; and a utilization factor between 0 and 1 (between 0% and 100%) will imply a corresponding bar occupying the left area of the block indicating the layer 1 1 1.

[0056] Fig. 3 shows an apparatus 300 (which may be an example of the apparatus 100 and / or 200) which shows the gating module 120 having a plurality of gating units 121 . Here, each layer 1 1 1 of the slimmable neural network processed by the NN processor 1 10 controls a layer 1 1 1 of the slim- mable neural network. For example, the layer 1 (initial layer) may be controlled by tire gating unit 1 through the utilization factor ut' (t indicating the segment index in the input sequence, e.g. time in- stant). The gating unit 1 receives as input the features fi1(intermediate result) which is in this case the input segment 102. The layer 2 is controlled through the utilization factor u,2by the gating unit 2. The gating unit 2 takes into consideration the features ft2as outputted by the layer 1 and / or the input segment 102. This can be reiterated for each layer, after the last layer M which is controlled through the utilization factor UtMby the last gating unit M. The gating unit M is controlled by the input segment 102 and / or by the feature ftMtaken from the layer M-l . In some examples, the input segment 102 may be avoided for at least one gating unit or for ail the gating units. It is also to be noted that it is not necessary that there is one single gating unit for one single layer: It may be that there is one gating unit for multiple layers (e.g., immediately adjacent layer, and / or there may be one single gating unit that deactivates only one part of one layer).

[0057] In Fig 3 it may be that the features ft' are the same as the input segment 102 or may be only a selection of the input segment 102. As can be seen from Fig. 3, the layer 1 has a small percentage of activated units (e.g., 30%) by virtue of the utilization factor u(* being small (e.g., 30%). The layer 2 has in this case a utilization factor Ui2which is 100% and, accordingly, no deactivatable unit is deactivated at all, and all the deactivatable units operate. In layer M, the majority (e.g. approximately 70%) of the deactivatable unit is activated, while only a small percentage (e.g., approximately 30%) of the activatable unit is deactivated. This may be because the utilization factor utMis approximately 70%.

[0058] Fig. 4A shows an example of a fully connected neural network with three neurons in the first neural layer, three neurons in the second neural layer and three units in the third (last) neural layer. Each neuron o f the first layer is connected (through a connection carrying a weight) with each neuron of the second neural layer. Each neuron of the second neural layer is connected (through a connection carrying a weight) with each neuron of the neural network. Each connection (connecting a first neu- ron to second neuron) may be understood as having a weight which, during inference, is applied to scale the output value from the first neuron, the weighted value being inputted into the second neuron. Therefore, from the first layer to the second layer, each neuron has three connections (each associated with a particular weight), to a total number of 9 weights. The same applies from the second layer to the third layer. The weights of the connections are learned during training and are used to scale the numerical values during inference. In Fig. 4A the deactivatable units are neurons.

[0059] As can be noted form Figs. 4B and 4C, however, the deactivation of a neuron also automat- ically causes the deactivation of the connections entering into and exiting from the neuron, thereby simplifying the computation.

[0060] In Fig. 4B, it is shown that the deactivatable unit 3 of the second layer is deactivated (e.g., by virtue of a decision of the gating module 120 and / or of the decision of the particular gating unit 121 controlling the second layer). As can be seen from Fig. 4B, during inference, the deactivatable unit 3 of the second layer is deactivated (visibly disappearing), and it is not necessary to perform the scaling of the weights associated to the connections inputted into the deactivatable unit 3 of the sec- ond layer. Accordingly, only six connections remain and the multiplications to be performed are only six in number from the first layer to the second layer. The same applies from the second layer to the third layer: There are only six connections because there are no connections outputted from the de- activatable unit 3 of the second layer. Notably, with respect to Fig. 4a, this results in a diminution of the number of connections (and multiplications to be performed) from 18 connections (and multipli- cations) to 12 connections (and multiplications). Therefore, an appreciable amount of computational power is not wasted. The situation of Fig. 4B may be that, for example, of Fig. 3.

[0061] Fig. 4C shows the same example of Fig. 4A in which, however, each neuronal layer is sub- jected to a deactivation of one single deactivatable unit. For this reason, there are only four connec- tions from the first layer to the second layer and from the second layer to the third layer, thereby resulting in a good reduction of computational power wasted. In this case, the utilization factor may be considered to be 66.66% for each layer and may correspond, for example, to the example of Fig. 2 (or for the example of Fig. 3 in the particular case in which each gating unit 121 chooses the same utilization factor).

[0062] Fig. 4D shows another example of a method which may be an example of Figs. 4A, 4B, 4C in which the first layer has one single neurons deactivated (utilization factor 66.66%), the second layer has all neurons activated (utilization factor of 100%) and the third layer has a deactivation factor of 66.66% (one single neuron being deactivated). This situation may be, for example, the sit- uation of Fig. 3.

[0063] Fig. 5 A shows an example 400 which may be, in a particular case, of any of the apparatuses 100, 200, 300 of Figs. 1 , 2, 3 above. Here, as can be seen, the gate 120 may be inputted with the additional auxiliary input signal 105.

[0064] The gating unit 120 may, therefore, also be controlled by the auxiliary input signal 105 to generate the deactivation command 122 to the layers 1 1 1 to be controlled. It is to be noted that, in this case, it is shown that the utilization factor is the same for all the layers (and, therefore, it would appear that we are in an embodiment of Fig. 2); however, it may be that the gating unit 120 applies different utilization factors to different layers 1 1 1 based at least in part on the additional input signal 105. Accordingly, it may be imagined that either in Fig. 2 or in Fig. 3 additional input signals 105 are also inputted on the gating unit 120 (or, in some cases, on each gating unit 121 or at least in some of the gating units 121 ).

[0065] A variant of the apparatus 400 of Fig. 5 A is the apparatus 500 of Fig. 5B (which may be an embodiment of any of the apparatuses 100, 200, 300, 400). In this case, the gating module 120 is controlled by the output 104 (C)t) provided from the neural network. In this case, it may be understood that the output 104 at the timing step t will condition the utilization factor ut+i in the command 122 at the time instant t+1 . Also in tins case, the apparatus 500 seems an embodiment of the apparatus 200 of Fig. 2 because the utilization factors appear to be the same. Notwithstanding, the output 104 may be used to control the gate 120 (and the utilization factor as well) and also in the example of Fig. 3 (or more in general in an example in which multiple utilization factors are applied to different layers).

[0066] Fig. 4E shows an example of a gating module 120 controlling multiple layers based on a signal inputted to the gating module for the classification. It may be understood that the signal may be the input signal 102 (like in Figs. 1 and 2) and / or the multiple signals 102 and li1, ft2, ..., f(M(intermediate output segment 123, 1 12) like in Fig. 3, and / or the auxiliary input signal 105 and / or the output 104.

[0067] Fig. 4F shows an example in which the gating unit 120 controls only the neurons of the third layer based on the intermediate output segment inputted to the gate unit 120. Here, the intermediate output segment 123 outputted from the second layer may basically constitute the intermediate fea- tures ft2of gating unit 2 of Fig. 3, for example.

[0068] In particular, Fig. 4F shows an intermediate output segment 123 inputted to the gating unit 120 which, in turn, controls the activation 125 (embodying instantiations of the command 122) based on the intermediate output segment 123.

[0069] In the Figs. 4A-4F the deactivatable units have always been considered to be the neurons, and the absence of the neurons caused the connection also being automatically activated or deac- tivated based on the deactivated or activated status of the related neuron. Accordingly, Fig. 6A ap- pears to be the same as Fig. 4A because, in the particular case of Fig. 6A, all the 18 connections are activated. However, it is possible to deactivate some connections. In this case, the connections may be activated through an apparatus similar to any of the apparatuses 100, 200, 300, 400 or 500 but, in this case, what is deactivated are only connections and not neurons. In some cases, the deactivatable units may be both connections and neurons.

[0070] It is now explained the criteria according to which the gating module 120 decides the command 122 and, in particular, that the utilization factor (or more in general decides which deactivatable unit to deactivate). The gating module 120 may (e.g. through a deterministic or learnable technique) classify the input segment 120 based at least on the input segment. The classification may be between: a first classification result, which is indicative of the input segment 102 requiring a pro- cessing at a first computational complexity (which may be a low computational complexity); and a second classification result, indicative of the input segment 102 requiring a processing at a second (e.g., higher) computational complexity.

[0071] The gating module 120 may be configured to deactivate a greater number of deactivatable units in the case of the first classification result than in the case of the second classification result. Therefore, if more computational complexity is determined as being necessary, then less deactivata- ble units are deactivated (for example, the second, higher computational complexity may be neces- sary for highly noisy signals) (e.g., low signal-to-noise ratio, noisy speech or more in general noisy signal). In the case of higher signal-to-noise ratio (e.g., clean speech in the case of audio signal) many deactivatable units may be deactivated (and therefore less weightings may be carried out, thereby reducing the computational waste). This may be valid for the whole NN, or for one single group of layers or for some groups of layers (e.g. some remaining layers or groups of layers may remain unmodified, while the deactivations may be only one single layer or group(s) of layers). The deac- tivated deactivatable units may be the lowest-ranking units.

[0072] This classification between the first classification result and the second classification result may be generalized in a classification between a plurality (e.g. more than 2) of classification results indicated of different computation complexities: The gating module 120 will deactivate a greater number of deactivatable units for an input segment having a classification result indicative of the input segment requiring a processing at comparatively low complexity than for an input segment having a classification result indicative of the input segment requiring a processing at comparatively higher computational complexity. This may be valid for the whole NN, or for one single group of layers or for some groups of layers (e.g. some remaining layers or groups of layers may remain unmodified, while the deactivations may be only one single layer or group(s) of layers). It may be understood that, the higher the computation complexity required, the higher the amount of deactivat- able units (and in particular of lower-ranking deactivatable units) is activated.

[0073] In addition or alternatively, the gating module 120 may classify the input segment 102 based at least on the output result (like in Fig. 5B) 104 (e.g., at the time instant t for the processing at the time instant t+1). For example, the gating module 120 may deactivate a greater number of deactivat- able units (e.g., at least for one layer or for at least one group of layers or for all the layers of the neural network) in case the first classification result than in the case of the second classification result (the second classification result requiring higher computational complexity). This may be general- ized not only between a first classification result and a second classification result but between a plurality of classification results in such a way that different numbers of deactivatable units are de- activated at a necessity of having different levels of computational complexity. This may be valid for the whole NN, or for one single group of layers or for some groups of layers (e.g. some remaining layers or groups of layers may remain unmodified, while the deactivations may be only one single layer or group(s) of layers).

[0074] In addition or alternatively, the gating module 120 may classify the input segment 102 based at least on an intermediate result (e.g. 1 12, 123, f1, ft2, ..., ftM... ,) (like in Fig. 3 and Fig. 4F) outputted by an intermediate layer (or by the layers of a group lo layers). The gating module 120 will deactivat e the greater number of the deactivatable units in case of a first classification result (indicative of a first complexity required, which is lower than the second complexity required indicated by the second classification result), and will deactivate a lower number of the deactivatable units (or deactivate no deactivatable unit at all) in case of the second classification result (indicative of a second complexity required, which is higher than the first complexity required indicated by the first classification result). This also can be generalized in more than two classification results so that different numbers of deactivations are caused by different classification results. This may be valid for the whole NN, or for one single group of layers or for some groups of layers (e.g. some remaining layers or groups of layers may remain unmodified, while the deactivations may be only one single layer or group(s) of layers).

[0075] The gating module 120 may classify the input segment 102 based at least on noisiness meas- urement of the input signal 102 (a processed version thereof can also be measured). A first classifi- cation result, indicative of comparatively low noisiness (clean speech for example) will imply that a higher amount of deactivatable units is to be deactivated (thereby reducing computational waste), while a second classification result, indicative of comparatively high noisiness (e.g., noisy speech) will imply a lower number (e.g., none at all) of deactivations of the activity values. This may be generalized in a greater number of classification results, so that higher amounts of deactivations are implied by lower noisy signals (cleaner speech). This may be valid for the whole NN, or for one single group of layers or for some groups of layers (e.g. some remaining layers or groups of layers may remain unmodified, while the deactivations may be only one single layer or group(s) of layers).

[0076] The classification by the gating module 120 may be based, in some examples (e.g. in exam- ples in which the input segment 102 is an audio signal), on a harmonicity measurement of the input signal 102 (or a processed version thereof). Therefore, there may be a first classification result, in- dicative of comparatively high harmonicity (e.g. clean speech) and a second classification result, indicative of comparatively low harmonicity (e.g. noisy speech). In this case, the gating module 120 may deactivate a greater number of deactivatable units in the case of comparatively high harmonicity than in the case of comparatively low harmonicity. The harmonicity measurement may be general- ized in a plurality of harmonicity measurements (in particular, more than 2) which cause a plurality of classifications (e.g. more than 2), which imply different numbers of deactivations (e.g., the lower the harmonicity, the lower the number of deactivations). This may be valid for the whole NN, or for one single group of layers or for some groups of layers (e.g. some remaining layers or groups of layers may remain unmodified, while the deactivations may be only one single layer or group(s) of layers).

[0077] Additionally or alternatively (e.g. in examples in which the input segment 102 is an audio signal), the gating module 120 may classify the input segment 102 based on an interference meas- urement between a first classification result, indicative of comparatively low interference, and a sec- ond classification result, indicative of comparatively high interference. The gating module 120 may therefore deactivate a greater number of deactivatable units in case of the comparatively low inter- ference than in case of the comparatively high interference. These results may be generalized for more than two classification results, so that the higher the interference, the lower the number of deactivations. This may be valid for the whole NN, or for one single group of layers or for some groups of layers (e.g. some remaining layers or groups of layers may remain unmodified, while the deactivations may be only one single layer or group(s) of layers).

[0078] Additionally or alternatively, the gating module 120 may classify the input segment 102 based at least on an additional (e.g. auxiliary) signal (e.g. 105), like in Fig. 5A. For example, there may be a first classification result, indicative of the input segment 102 requiring a processing at the first, lower computational complexity, and a second classification result, indicative of the input seg- ment 102 requiring a processing at the second computational complexity which is higher than the first computational complexity. This result may be generalized for more than two classification re- sults, thereby implying multiple levels of deactivations. For example, the input segment 102 may be an audio signal (and the neural network may be used to separate a speech component from a noise component, or otherwise to process the input audio segment 102) while the additional signal (auxil- iary signal) 105 may be a video of the lip movement. The video signal 105 may be synchronized with the input audio signal 102 so that a relationship between the images and the audio can be performed. In this way, the gating module 120 may predict, from the lip movement and the input segment 102, the power of the target speech relative to the power of any interfering audio signal in the input segment 102, thereby operating a classification of the speech based on prediction. Based on the clas- sification, the gating module 120 will decide how many deactivatable units to deactivate. For exam- ple, in case that the gating module 120 infers, from the lip movement, that the speaker is not currently speaking at the time instant t, the input signal at instant t will be interpreted as noise or silence and the number of deactivations may be increased for instant t. In addition, in case the lip movement at instant t movement permits to understand that the input signal is currently speech, then the number of deactivations may be higher or lower for instant t, e.g. depending on the relative power of the target to the interferer(s), where the number of deactivations is higher when the power of the target is higher and the number of deactivations is lower when the power of the target is lower.

[0079] In another case, with an additional auxiliary signal 105 being, this time, an audio signal and the input segment 102 being an audio signal, the additional auxiliary audio signal 105 may be an input segment with the same voice of the speaker so as to classify the input segment 102 between a first classification result indicative of the input segment at instant t having tire voice of the speaker and a second classification result indicati ve of the input segment at instant t not having the voice of the speaker. For example, in case it is recognized that the voice of the speaker is prevalent in the input segment 102, then more deactivatable units are deactivated, while in the case that it is recog- nized that the speaker’s voice is not present in the input segment 102, the number of deactivations is reduced.

[0080] In the examples above, for the sake of explanatory easiness it is normally imagined that the deactivations are in one layer or in a group of layers or in the totality of the layers. It may be imagined that only one layer or group of layers (but in some cases not the totality or layers) is subjected to deactivations, while the remaining layers do not participate to any deactivation. However, it is also possible to imagine that the one layer or group of layers (but not the totality or layers) is subjected to a first type of deactivations, while another layer or group of layers (e.g. the layers which complement to the totality or layers) is subjected to a second type of deactivations, different from the first type of deactivations. The first and second types of deactivations may be based, in some examples, on the same criteria or on criteria which are similar or which provide information based on a different met- rics. For example, a first layer or group of layer could be subjected to a first type of deactivations based on a first classification, and a second layer or group of layer could be subjected to a first type of deactivations based on a second classification (where the second classification may be different from the first classification). In some examples, both the first classification and the second classifi- cation measure the complexity required, but in some examples based on different input values and / or different measurements on the same input values or on different input values. An example is in Fig. 3, where different layers are subjected to different gating units 121 . However, in addition or alterna- tive the selection may be based on different criteria. E.g. “gate 1” in Fig. 3 could be based on meas- uring signal-to-noise ratio (e.g. with the above-explained first classification result and second classi- fication result), while “gate 2” could be based on measuring harmonicity (e.g. with the above-ex- plained first classification result and second classification result). It is however possible that the same criterion is used for all the layers or groups of layers of the NN. In other cases, it is possible to localize the deactivations in one single layer or group of layers, while mainlining the remaining layers un- changed.

[0081] Discussion

[0082] Proposed Method and technique

[0083] We propose a dynamic SNN (e.g. as shown in Figs. 1 -6B) for processing sequential data, which can dynamically adapt its computational graph based on the characteristics of the input data points ( 102, It). The proposed method consists of a SNN 5 (e.g. processed by the NN processor 1 10) and a gating module 120, represented mathematically as t), as de- picted in Figs. 1 -3, 4E, 4F, 5A, 5B, etc.).

[0084] To illustrate, let Itdenote a segment 102 (i.e., one or more data points) of the input sequence to the proposed method, where t is the index (e.g. time) of the segment in the sequence. And let Otrepresent the output 104 of the proposed method, which can either be continuous or discrete. In the following, we describe the two blocks of the proposed method in more detail.

[0085] Slinimable neural network The SNN S (processed by the NN processor 1 10) may consist of (or in particular comprise) M slimmable layers Slfor L = 1,2, ... , M , that can be executed at different widths for computing the output (for example, in Fig. 4A the first layer with its three neurons is 51, the second layer with its three neurons is S2, the third layer with its three neurons isS3). For each slimmable layer Sl, its weight parameters may be slimmed given a utilization factor ulG [0,1] (or a percentage, or an absolute value), where u = 1 corresponds to full utilization of the layer’s parameters (100%), and u = 0 corresponds to skipping the layer and relaying its input to the output (0%) (as already explained, in Fig. 4C we have u — 0.666666, or 66.66%). One possible approach to slim the weight parameters of a SNN is slicing. Using MATLAB-like notations for array slicing, a slimmable feedforward layer, for instance, with a weight tensor WlG RD°xD‘ (e.g. in Fig. 4A: W1E R3x3and 14 / 2G R3x3, where W1has the 9 weights connecting the three neurons of the first layer with the three layers of the second layer, and 14 / 2has the 9 weights connecting the three neurons of the second layer with the three layers of the third layer) is sliced with a utilization factor u as Wl[1: u x Do, : ] (e.g. in Fig. 4C, with u = 2 / 3 we have 3, : j = and Dodenotes the input and output dimension (in Figs. 4A and 4C being Dj = 3 and Do— 3 for each layer), respectively. Using mathematical nota- tions, slicing the weight tensor 14 / 1results in a new tensor with elements (l4 / / (in Fig. 4C, (kl / ,-1,) ), where i and / are the indexes of thev (Uyl<i<uxD0;l<j<Dj °v' l<i<2 / 3x3;l<j<3Jrows and columns of Wl, respectively. As can be understood, the lower the index i, the higher the ranking of the weight.

[0086] Gating module

[0087] To enable the SNN to dynamically adapt its layer widths based on the input segment 102, the gating module 120 is employed in the proposed method. The gating module Q ( 120) may predict the required utilization factor(s) by performing a classification on the input seg- ment 102 and / or some intermediate outputs of some levels and / or additional (auxiliary inputs 105) and / or the output 104 in feedback. The gating module 120 may therefore analyze the characteristics of the input segment 120 and / or some intermediate features from the SNN. The output uti lization factor(s) u (or ut) (in the command 122) of the gating module 120 can be either continuous-valued (i.e., regression) or discrete-valued (i.e., pre-defined classes). One example of a gating module 120 may be a neural network, whose parameters are opti- mized based on speci fic criteria, for example, to tradeoff between efficiency (e.g., computa- tional cost) and performance (e.g., accuracy). Two possible gating mechanisms can be used for implementing the proposed method: Global gating and local gating, illustrated in Fig. 2 and Fig. 3, respectively.

[0088] Global gating

[0089] In this scheme, the gating module has a single gating unit (e.g., one single neural network) which is utilized to globally control the width of all slimmable layers of the SNN, as depicted in Figs. 2, 5A, 5B. This can be mathematically formulated as

[0090] Another possible implementation of the global gating scheme is to compute a differ- ent utilization factor for the different slimmable layers 11 1 , i.e.,

[0091] Figure 2: Proposed method utilizing a global gating mechanism, where a single gate controls the width of all slimmable layers.

[0092] Local gating

[0093] In this scheme (like in Figs. 2 and 4F), there are M different gates (each being indicated with 1 1 1) associated with the slimmable layers Slof the SNN, each predicting a utilization factor ulfor the corresponding slimmable layer (with I between 1 and M; for example, in Fig. 4F M=3). Denoting the intermediate features at the input of the / -th slim- mable layer by ftL, the / -th gate determines the utilization factor as utl= 9l(.ft 9 (3)

[0094] Optionally (like in Fig. 3), the input segment / t(120) to the SNN may also be pro- vided as an additional input to the gates, i.e., 4 =^ 9\fLit)- (4)

[0095] Figure 3: Proposed method utilizing a local gating mechanism. An independent gate for each slimmable layer in the SNN controlling its width by analyzing the intermediate features ftl. Optionally, the input segment ltcan be provided to the gates as an additional input.

[0096] APPLICATIONS

[0097] As the proposed method is designed to dynamically adapt its computational cost based on the characteristics of input segments of a sequence, it offers potential advantages for a wide range of applications, especially when applied on resource-constrained edge devices (e.g., smartphones). Ex- amples of these applications are mentioned in the following.

[0098] Speech enhancement (SE): Generally speaking, SE aims to improve the quality and intel- ligibility of speech signals by removing undesired audio signals (e.g., noise). SE encompasses several tasks, for example:

[0099] • Noise reduction (NR): A NR system aims to enhance a desired speech signal (i.e., from a single speaker) contaminated with an undesired non-speech signal (e.g., fan noise, sensor noise, etc.). A typical NR system accepts an audio signal as input and outputs an enhanced audio signal.

[0100] • Target speaker extraction (TSE): A TSE system aims to enhance the speech of desired speaker and considers other audio sources to be undesired (e.g., interfering speakers, non-speech noise). To specify the target speaker in a TSE system, an auxiliary signal is provided (e.g., speech snippet from the target speaker, a speaker embedding encoding the voiceprint of the target speaker, visual input containing the target speaker’s lip movement). In this case, a TSE accepts an audio signal as well as an auxiliary signal (or more than one auxiliary signal) and outputs an enhanced audio signal.

[0101] • Speech separation (SS): A SS system typically aims to isolate the speech signals of all speakers in an overlapped speech signal. In this case, a SS system accepts an input audio signal and output many audio signals corresponding to each speaker in the input signal.

[0102] SE systems typically process the input signal in a block-wise fashion (e.g., time frame-level). The proposed method can be applied in SE to allow adapting its computational cost to the difficulty of the input block, thus reducing unnecessary computat ions for ’easy’ input blocks, e.g., with silence or high signal-to-noise ratio. The computational efficiency offered by the proposed method is partic- ularly beneficial when SE systems are deployed to resource-constrained edge device, such as smartphones, hearing aids, etc. One embodiment of the proposed method is described in more detail in the (unpublished) paper attached to this report.

[0103] Some aspects

[0104] Below are a list of aspects.

[0105] 1. A sequential data processing method or apparatus that comprise (or consists) of the following (see Fig. 1 ):

[0106] - A slimmable neural network (SNN) (or a SNN processor 1 10) operating on an input segment 102 to produce an output 104, where the computational complexity of the network is controlled by a set of utilization factors (e.g. in command 122) that determine the proportion of the parameters of the layers in the SNN used for output computation;

[0107] - A gating mechanism (e.g. in 120) controlling the SNN by generating a set of utilization fac- tors) (e.g. in command 122) from the input segment(s) 102 and / or features derived from the input segment(s) (e.g. intermediate outputs from the layers).

[0108] 2. A method or apparatus according to Aspect 1 operating on audio signals, in which the gating mechanism generates time-dependent utilization factor(s).

[0109] 3. A method or apparatus according to Aspect 1 or 2, in which the gating mechanism generates discrete utilization factor(s).

[0110] 4. A method or apparatus according to any of Aspects 1 -3, that can accept additional inputs (e.g., 105, such as a speaker embedding in a target speaker extraction application).

[0111] 5. A method or apparatus according any of Aspects 1 -4, which can provide additional outputs (e.g., reconstructing all audio sources in a source separation application).

[0112] 6. A method or apparatus according to any of the previous Aspects performing speech enhance- ment.

[0113] In examples above and below, there are advantages in deactivating any unit (neurons, pat- terns, connections, biases, input elements, etc.): in correspondence to the deactivations, no calcula- tion is to be carried out (and therefore calculation amount is reduced and power consumption is reduced and the computation result is achieved more quickly).

[0114] Some specific impl emeu I at i oi i s

[0115] Neural networks for speech separation generally exhibit high computational costs and large memory footprints. Moreover, typical separation networks have a fixed computation graph that pro- cesses all input frames at a uniform computational cost, even though intensive processing might not be necessary for frames with silence or one active speaker. This computational inefficiency becomes particularly pronounced in resource-constrained devices, such as edge devices. In this letter, we pro- pose the dynamic slimmable network (DSN) (e.g. processed by NN processor 1 10) for speech sepa- ration that mitigates the computational inefficiency of existing networks. We introduce slimmable layers (or more in general slimmable groups) with a gating mechanism (e.g. controlled by 120) that can adapt the computation graph based on the input signal characteristics. As an example, we propose to use the slimmable layers in the intra-chunk blocks of a dual-path structure-based network to facil- itate adaptation based on the signal's local characteristics. Experimental evaluation on simulated two- speaker mixtures from the WSJ0-2mix dataset demonstrates that the proposed method substantially reduces the computational cost while maintaining the performance in comparison to the fully utilized static networks.

[0116] Deep neural networks (DNNs) have shown remarkable performance for monaural speech separation in recent years

[0001] -[5]. However, existing separation networks are characterized by high computational demands and substantial memory requirements [6]. A considerable source of the com- putational inefficiency of typical separation networks stems from their static nature, where all input frames are processed with a uniform computational cost, even though there might be frames com- prising non-overlapping speech or silence that do not require intensive computations. These limita- tions are particularly pronounced when the separation networks are deployed on resource-constrained devices, e.g., edge devices. Therefore, it is of interest to develop a separation network that can dy- namically adapt its computational graph based on the input signal characteristics, ensuring efficient use of computational resources.

[0117] Conditional computation [7]-[9] (a.k.a. dynamic inference

[0010] ,

[0011] ) is a promising ap- proach to improve the computational efficiency of DNNs at run-time by selectively utilizing a subset of the network in an input-dependent fashion. The framework encompasses a plethora of techniques, including early exiting

[0012] , mixture-o (-experts (MoEs)

[0013] ,

[0014] , slimmable layers

[0015] ,

[0016] , to name a few. Early exiting allows a network to stop execution at an intermediate layer during infer- ence based on a particular criterion, and therefore reduce unnecessary computations. MoEs comprise independent sub-networks (i.e., experts) and gating modules that dynamically route the inputs to such experts. Although some variants of MoEs (e.g.,

[0017] ) can increase the model complexity at the same computational cost, they typically suffer from large memory footprints, especially when the number of experts is large

[0018] . Slimmable neural networks (SNNs)

[0019] ,

[0020] , and its dynamic variants

[0015] ,

[0016] , enable efficient computation by slimming (or slicing) the weights of some of its layers at run- time. In principle, SNNs offer a more suitable solution than MoEs for efficient inference on edge devices since the computational complexity can be reduced while having a constant number of pa- rameters.

[0118] Several studies have explored dynamic inference for speech separation. For instance, the work in

[0021] employed an early exiting mechanism, which is triggered when two consecutive layers produce sufficiently similar predictions, whereas a gating module in

[0022] predicts whether to stop or continue execution at the beginning of each block. However, both methods

[0021] ,

[0022] apply dynamic inference on an utterance-level, rather than on a more granular frame-level basis, limiting the poten- tial for achieving higher computational efficiency. In

[0023] , a sparsely-activated MoEs was employed, in which the inputs are dynamically routed to distinct experts depending on the input characteristics. In a recent study

[0024] , we explored SNNs for speech separation and demonstrated its effectiveness in trading off performance with computational complexity. However, selecting the utilization factor at run-time was carried out based on the resource budget at run-time rather than on the characteristics of the input frames.

[0119] In this document, we present inter alia a dynamic slimmable network (DSN) e.g. for speech separation capable of modifying its computation graph at run time in an input-dependent fashion. A core contribution of this work is the design of a dynamic slimmable block (DSB) (e.g. in 1 10) that comprises a slimmable sub-network and a gating module 120 that e.g. predicts the necessary utiliza- tion factor based on the characteristics of its input (e.g. performing a classification). By incorporating the proposed DSB in the intra-chunk blocks of the SepFormer architecture [4], we demonstrate dy- namic inference on the short segment level, enabling a substantial reduction in computational com- plexity without a degradation in the separation quality.

[0120] Figure 7A: Block diagram of the proposed dynamic slimmable network (DSN) for speech separation. Utilizing the SepFormer architecture [4], we propose to replace the positional feed-for- ward network (FFW) in the intra-chunk transformers with the dynamic slimmable block (DSB). This allows dynamic inference at the chunk level, where a utilization factor u of the slimmable FFW is predicted for each chunk. Note that the inter-chunk transformers comprise a static FFW. For brevity, the residual path and the positional encoding are omitted in the block diagram of the intra-chunk transformer.

[0121] System Overview

[0122] Let y E IK7represent a single-channel signal of length T that comprises overlapped speech of / speakers, i.e., y = denotes the speech signal of the / -th speaker. Speech separation is carried out using a DNN (e.g. processed by NN processor 1 10), represented by IF, which reconstructs the speech of the constituent speakers given the ob- served microphone signal y, i.e., {sy}j= 1= .'F(y), where Sj is the estimated signal of the / - th speaker.

[0123] In this study, we demonstrate the potential of dynamic inference for speech separa- tion using the SepFormer architecture [4] as an example, depicted in Figure 7A. Utilizing the dual-path framework [ 1 J, SepFormer comprises intra-chunk and inter-chunk transformer blocks that model local and global characteristics of the input signal, respectively. We pro- pose to apply dynamic inference in the intra-chunk transformer blocks by analyzing the short-term information within a chunk, such as partial overlap or silence. Specifically, we replace the positional feed-forward network (FFW) in the intra-chunk transformer blocks with the proposed DSB, due to the substantial computational demand of the FFWs. It should be mentioned, however, that the DSB can be incorporated to other layers as well, e.g., by slimming the number of kernels in convolutional layers. In the following, we describe the DSB in more detail.

[0124] Dynamic Slimmable Block (DSB)

[0125] To illustrate, let Il,mG RFxCand Ol,mG RFxCdenote the input 102 and output 104 features of the DSB at the intra-chunk block indexed by I G {1,2, where m G {1,2, ... , M] is the chunk index, F represents the dimensionality of the features, and C denotes the chunk size. In SepFormer, the FFW may include (e.g. consist of) two (or more in some examples) linear layers and a ReLU activation in between. Omitting the biases for ease of exposition, the FFW at the / -th intra-chunk block is parameterized by the weight tensors G RDxFand I / F2 G RF xD, where D denotes the number of units in the hidden layer. The DSB may include (e.g. consist of) two (or more in some examples) sub-blocks: a slimmable sub-network and a gating module. In the following, we describe both sub-blocks in more detail.

[0126] Slimmable sub-network: Denoted by S, this sub-block may include (e.g. consist of) a neural network capable of being utilized using different subsets of its computation graph by varying the widths of its layers, i.e., number of units or channels, and thus enables saving computations at run-time [16, 19, 20, 24]. The degree of slimming, i.e., the fraction of the sub-network’s width to be utilized, can be specified by an additional input, referred to as a utilization factor u G (0,1]. In this study, we use a discrete set of pre-defined utilizations, denoted by 'll. We turn the static FFW in the intra-chunk blocks into a slimmable FFW by slimming the weights W3land 1 / F2Z, as where W[ = \ / V[ [1: u ■ = W2 [: ,1: u ■ D ]. To enable the slimmable sub-network to adapt its width based on the input features, a gating module is employed in the DSB, which predicts a utilization factor ul,mper chunk.

[0127] Gating module: Given the input features Il,m, the gating module Q predicts a prob- ability vector Pl,mG representing the probability scores corresponding to each utili- zation factor, i.e., Pl,m= Qpl,m). Following [ 17, 25], the gating module comprises a single feed-forward layer, parameterized by WglG Rllzlx2F, followed by a softmax activation. As input to this layer, we concatenate the temporal mean g.l,mG RFx land standard deviation al,mG RFX 1, computed by aggregating the input features within a chunk, i.e.,

[0128] Pl,m= softmax(r • Wgl• where ® is the concatenation operator, gl,m= , c], aL,m= an(j pieparameter T controls the sharpness of the softmax probabilities. For T > 1, the output probabilities are more polarized, making the softmax behave as a 'hard max’, while r < 1 leads to a more uniform distribution.

[0129] During training, the output of the DSB is computed by linearly combining the outputs of all different utilizations of the slimmable sub-networks weighted by the corresponding gating module's probability, i.e.,

[0130] This soft gating avoids the training instabilities associated with hard assignments [?]. During the inference phase, however, only the utilization corresponding to the highest prob- ability is selected to save computations, i.e., Optimization

[0131] The proposed DSN is optimized via a weighted sum of three loss functions: a signal reconstruction loss, a gate sparsity loss, and an importance loss. The signal reconstruction loss is computed using the scale-invariant source-to-distortion ratio (SI-SDR)

[0026] and the utterance-wise permutation-invariant training (uPlT)

[0027] , i.e., where Q is the set of all possible permutations of the ground-truth signals. Inspired by

[0014] , a sparsity loss is utilized to encourage sparse predictions of the gating modules and thus allow for hard gating during inference, i.e., where Pk'mis the -normalized probability vector. We propose to add the factor wk— uk / X / <uk m (16) to penalize having larger utilization factors and reduce the computational complexity. Furthermore, to diversify the outputs of the gating modules, we use a modified version of the importance loss, introduced in

[0025] , Here, we define the importance as the sum of the gating modules' predictions for each input example, i.e.,

[0132] The importance loss is then equal to the square of the coefficient of variation (C V) of the importance, i.e.,

[0133] The overall loss is then given by importance ,(19) where a and / ? are the weighting factors for the sparsity and importance losses, respectively.

[0134] Experimental Setup

[0135] Dataset: For our experiments, we used the WSJ0-2mix dataset

[0028] , which consists of 20k, 5k, and 3k two-speaker mixtures for training, validation, and testing, respectively. The mixing source-to-interference ratios were sampled from 0 dB to 5 dB. The signals were sampled at 8 kHz. We used the 'min' version of the dataset for training and evaluation.

[0136] Figure 7C: Performance of the proposed dynamic slimmable network (DSN) for different configurations compared to two static SepFormer networks. For the DSN models, the marker color is unique for each value of a. The parameter counts of the models are 9.89M, 13.04M, and 13.05M for Static (u = 0.25), Static (it = 1.0), and DSN, respectively.

[0137] Implementation: Following the notations in [4], the hyperparameters of the SepFormer ar- chitecture were set as follows: N = 2, A / intra= 4, / Vinter= 4, Heads = 8, and C = 50 (75 ms). (Since dynamic inference is carried out on the chunk level, we chose a rather smaller chunk size than the one used in the SepFormer paper (C=250). In our preliminary experiments, we found that a static SepFormer using either chunk size yields similar performance for the dataset used in this study). For the FFW in the intra- and inter-chunk transformers, the number of units in the hidden layer of the FFW was set to D = 1024. As mentioned in Section 2.1 , the FFWs in the intra-chunk blocks are allowed to be slimmed based on the predicted utilization factors from the gating modules. We used a pre-defined utilization set of 11 = {0.125,1.0}. The time-domain encoder and decoder had a kernel size of 16 samples with a hop size of 8 samples. The number of convolutional filters F in the encoder was set to 256.

[0138] Training: Adam

[0029] optimizer was used with an initial learning rate of 1.5 x 10“4. A scheduler was utilized to reduce the learning rate by a factor of 2 if no improvement of the validation loss was observed in three consecutive epochs. Early stopping with a patience of 6 epochs was ap- plied. The maximum number of epochs was set to 200. We used a batch size of 2 samples. The training examples were cropped to a length of 4 seconds. No dynamic mixing was applied. The gradients were clipped if their f2-norm exceeded a value of 5.

[0139] Figure 7B: Visualization of the gating modules' predictions, i.e., p / '”1, for the intra-chunk blocks I G {1,2, ... ,8 } (top), and the corresponding overlayed spectrograms of the two speakers in the mixture (bottom).

[0140] Performance Evaluation For evaluation, the SI-SDR improvement (Sl-SDRi)

[0026] is used to quantify the quality of the separated signals. In addition, we report the number of floating point operations (NFLOPs) as a measure for computational complexity (Computing the NFLOPs was carried out using the code from: https: / / github.com / zhijian-liu / torchprofile).

[0141] DSN vs. Static Networks

[0142] The performance of the proposed DSN is compared with two different static SepFormer models with / Vintra= 128 and A / intra= 1024, equivalent to a utilization of u = 0.125 and u = 1.0, respectively. We also explore the impact of the weighting factors a and / ? as well as the sharpness parameter T on the performance of the DSN. The results are shown in Figure 7C. For each model, we report the average SI-SDRi score and the average of the average utilization per test example. We first confirm that the DSN model trained without the auxiliary losses (a = 0.0, / ? = 0.0, T = 3.0) collapsed to a static network, where its gating modules always predict the same values irrespective of the input chunk. In contrast, the DSN models trained with auxiliary losses exhibit a dynamic be- havior for their gating modules with respect to the characteristics of the input chunk. This is espe- cially effective when both auxiliary losses are combined, as shown for the DSN model (a = 0.1, / ? — 0.1, T = 3.0), where the model is able to significantly reduce the average computational cost without incurring a substantial performance degradation, compared to the Static (u = 1.0). It is worth men- tioning that the average computational cost of the DSN model is highly signal-dependent and the test examples contain mostly overlapped speech. Furthermore, it is evident that selecting a large value of the sharpness parameter T is essential for the proposed DSN. This is expected as a large T polarizes the predict ions of the gating modules, resulting in a reduced mismatch between so ft and hard assign- ments in the training and testing phases, in (13) and (14), respectively.

[0143] Figure 7B shows the predictions of the gating modules of the proposed DSN model (a = 0.1, p = 0.1, T = 3.0) for a test example. The corresponding overlayed spectrograms of the two speakers in the input mixture are also depicted. As hypothesized, it can be seen that lower utilizations (u = 0.125) are predicted for silent frames, while frames with concurrent speech activity are pro- cessed with full utilization (u = 1.0). It is also interesting to observe that frames with a single active speaker tend to be processed with less utilization than the frames with concurrent speech activity. This highlights the potential of the proposed DSN to increase the computational efficiency via dy- namic inference on the chunk level.

[0144] DSN vs. MoEs

[0145] The DSBs in the proposed DSN can be viewed as a MoE layer but with experts sharing weights, i.e., subsets of each other. In this case, the gating module selects a subset of the slimmable sub-network as an expert via weight slicing. In terms of memory footprint, a DSB is better than a MoE layer due to weight sharing. However, it is not yet clear which paradigm provides better per- formance. Table 1 shows a comparison between the proposed DSN and two MoE models. The MoEs model (D = {128,1024}) has two experts with different sizes for the FFW in each intra-chunk block, whereas the MoEs model (D = {1024,1024}) has two experts with the same size. For a fair compar- ison, both MoEs models are trained with the same auxiliary losses as the proposed DSN. It can be seen from the results that the proposed DSN achieves better separation than the equivalent MoEs model (D = {128,1024}) while requiring a comparable number of computations. Even for the bigger MoEs model (D = {1024,1024}), which has a higher parameter count and requires substantially more computations, no significant performance gain can be observed compared to the proposed DSN.

[0146] 'fable 1 : Performance of the proposed dynamic slimmable network (DSN) compared to mixture-of- experts (MoEs)

[0147] Discussion

[0148] We proposed a dynamic slimmable network (DSN) for speech separation comprising layers that can be dynamically slimmed at run-time based on the characteristics of the input signal. Utilizing the SepFormer architecture, dynamic inference on the chunk level is achieved by turning the com- putationally intensive positional-feed forward layers in the intra-chunk transformers into dynamic slimmable layers. As hypothesized, the proposed DSN was able to track the speech activity of the input signals, allocating fewer computations for silent or non-overlapping frames and more for fully overlapped frames. This enables a higher computational efficiency compared to the coarse utterance- level dynamic inference, without degrading the separation performance.

[0149] Other examples

[0150] Here, different inventive examples, embodiments and aspects are described. Also, further embodiments will be defined by the enclosed claims. It should be noted that any embodiments as defined by the claims can be supplemented by any of the details (features and functionalities) de- scribed in the following chapters. Also the features defined in square brackets in the claims and / or in round brackets define new embodiments which are optional features.

[0151] Also, the embodiments described in the following and / or previous chapters can be used in- dividually, and can also be supplemented by any of the features in another chapter, or by any feature included in the claims. Also, it should be noted that individual aspects described herein can be used individually or in combination. Thus, details can be added to each of said individual aspects without adding details to another one of said aspects. It should also be noted that the present disclosure de- scribes, explicit ly or implicitly, features of a mobile communication device and of a receiver and of a mobile communication system. Thus, any of the features described herein can be used in the context of a mobile communication device and in the context of a mobile communication system (e.g. com- prising a satellite). Therefore, disclosed techniques are suitable for all fixed satellite services (FSS) and mobile satellite services (MSS). Moreover, features and functionalities disclosed herein relating to a method can also be used in an apparatus. Furthermore, any features and functionalities disclosed herein with respect to an apparatus can also be used in a corresponding method. In other words, the methods disclosed herein can be supplemented by any of the features and functionalities described with respect to the apparatuses. Also, any of the features and functionalities described herein can be implemented in hardware or in software, or using a combination of hardware and software, as de- scribed below. Depending on certain implementation requirements, examples may be implemented in hardware. The implementation may be performed using a digital storage medium, for example a floppy disk, a Digital Versatile Disc (DVD), a Blu-Ray (registered trademark) Disc, a Compact Disc (CD), a Read-only Memory (ROM), a Programmable Read-only Memory (PROM), an Erasable and Programmable Read-only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM) or a flash memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable. Generally, examples may be implemented as a computer program product with program instructions, the program instructions being operative for performing one of the methods when the computer program product runs on a computer. The program instructions may for example be stored on a machine-readable medium. Other examples comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier. In other words, an example of method is, therefore, a computer program having a program-instructions for performing one of the met hods described herein, when the computer program runs on a computer. A further example of the methods is, therefore, a data carrier medium (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier medium, the digital storage medium or the recorded medium are tangible and / or non-transitionary, rather than signals which are intangible and transitory. A further example comprises a processing unit, for example a computer, or a programmable logic device per- forming one of the methods described herein. A further example comprises a computer having in- stalled thereon the computer program for performing one of the methods described herein. A further example comprises an apparatus or a system transferring (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver. In some examples, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some exam- ples, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods may be performed by any appropriate hard- ware apparatus. The above described examples arc illustrative for the principles discussed above. It is understood that modifications and variations of the arrangements and the details described herein will be apparent. It is the intent, therefore, to be limited by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the examples herein.

[0152] Concepts

[0153] Concepts above are summarized here below.

[0154] In accordance to a concept, there is provided an apparatus [e.g. a signal processor apparatus; the apparatus may be for speech enhancement (e.g. noise reduction, target speaker extraction or more in general target component extraction e.g. speech separation; the apparatus may be for media enhancement and / or video enhancement] comprising: an input interface to receive an input segment [e.g. I] of an input sequential data [e.g. an input sequential signal, such as in time domain (e.g. having a sequence of time domain samples) or in frequency domain (e.g., having a time-based sequence of time-frequency bins)), or in position domain (such as text, which provides information on speech e.g. based on the position of a phoneme)] [e.g. an audio signal (e.g. evolving in time), or a video signal (e.g., evolving in time, e.g. as a sequence of video images), or more in general a media signal (e.g. having audio signals and video signals combined with each other] [e.g., in some examples, the sequential data may be a text in the time domain or in the position domain][the input sequential data may evolve, e.g. monodimension- ally, along an ordered domain (e.g. time, space, etc.)] a neural network, NN, processor to derive an output result [e.g. a processed version of the input segment (e.g. a voice component (or another target component) of the input segment in the case the input signal being an audio signal being a mixture of speech and noise), and / or a noise- reduced version an audio input signal, and / or a speaker component (or more in general target component) of an audio input signal; or a compressed (e.g. encoded) or decompressed (e.g. decoded) version of the input segment or a classification result on the input segment] by processing the input segment [e.g., I] through a NN [e.g. a slimmable NN] having a number of layers [e.g., at least 2 layers] from a first layer [e.g, layer 1, e.g. as in Figs. 2 and 3] to a last layer [e.g. layer M, e.g. as in Figs. 2 and 3] [e.g. with the possible interposition of at least one hidden layer between the first layer and the last layer}, the NN using a predetermined number of deactivatable units which are selectively deactivatable [e.g. the deactivatable units may be (e.g. in the case of fully connected NNs) neurons, connections between neurons (e.g. each connection having a weight), biases, input elements), learnable parameters of the NN, while the deactivatable units may be (e.g. in the case of convolutional NNs) patterns, for example] [e.g., in accordance with the particular example, some units are deactivatable: for example, in one embodiment of a fully connected NN, the deactivatable units may be (al! or at least some of) the neurons; in some examples, deactivating one or more neurons may implicate that also the entering connections are deactivated, and the scaling of the input values by the weights are therefore skipped, thereby reducing the computational effort; in some examples, the deactivatable units may be (all or at least some of) the connections but not necessarily the neurons, and the deactivation of a connection may implicate that some scaling of input values to some neurons are not processed, thereby reducing the computational effort] [ e.g., other types of deactivatable units are patterns (e.g. in convolutional NNs), biases (whose deactivation may therefore save from adding the bias value to the calculated values), and input elements (whose deactivation may therefore result in reducing the dimension of the input segment)]] e.g., any combination of deactivatable neuron, patterns, connections, biases, and input elements is admitted] [e.g. deactivatable units can be at least one of some of neurons, patterns, learnable parameters of the NN, connections (e.g. associated with weights, biases, input elements)} [e.g., each layer may have or be connected through a specific number of deactivatable units (e.g. neurons, patterns) and the specific number of'deactivatable units may be the same or different from a predetermined number of deactivatable units of at least, one other / ayer][ e.g., in some, examples (e.g. in a fully-connected NN), each deactivatable unit may be a neuron, and each neuron of each layer (apart, from the last layer) may be connected with each neuron of the immediately preceding layer through a learnable neuron-to-neuron specific weight, and each layer (apart from the first, layer) may be connected with each neuron of the immediately preceding layer through a learnable neuron-to-neuron specific weight, e.g. in the case of a fully connected NN][ e.g., in some examples (e.g. in a convolutional NN, CNN), each deactivatable unit may be a specific predetermined pattern (filter) to be convoluted along the input segment (the input segment may be, in this case, a tensor (e.g. a multi-dimensional tensor, such as a 2-dimensional tensor or 3-dimensional tensor or having more than 3 dimensions), to provide a tensor result (e.g. the tensor result being obtained by multiply the values of the pattern by the values of the input segment)]} [e.g., in some examples, deactivatable units may include connections between neurons (e.g., associated with weights)] [e.g., in some examples, deactivatable units may include bias values] [e.g., in some examples, deactivatable units may include connections input elements (e.g., elements of the input seg- nient)] [ e.g., therefore, in some examples the deactivatable units may include neurons, patterns, weights, biases, learnable parameters of the. NN, input elements in any combination] {e.g., in some examples, for each layer the deactivatable units are ranked from a first-ranked deactivatable unit to a last-ranked deactivatable i / m7][ e.g., the output of the NN processor may be, in some examples, sequential, e.g. there may be one output value [e.g., scalar or tensorial (e.g. vectorial, matricial, 3- D-tensorial, etc.) for each value of the ordered domain (e.g. time, space, etc.); in other examples, the output value may be an integral feature of the input segment (e.g. a value that provides information on multiple values of the ordered domain, e.g. a value that provides information on the entire input segment, such as a time period in the case the ordered domain is time, or a spatial length (or positional length) in the case the ordered domain is a space (or a position))]},' a gating module [e.g. learnable gating module] [the gating module may include a plurality of single gating units, e.g. like in Fig. 3, each single gating unit being associated with one layer or a plurality of layers] configured to deactivate at least one deactivatable unit [e.g. the gating module may deactivate a number of deactivatable unit between 1 and all the deactivatable units; in some examples the gating module may be configured to select between deactivating no deactivatable unit at all and deactivating the at least one deactivatable unit, e .g. based on a classification result] based on the input segment and / or on at least one intermediate output segment [e.g. an output from the first layer or an output from any hidden / ayer][e.g., the gating module may be configured to deactivate the at least one deactivatable unit based, on the input segment or based on the intermediate output segment outputted by the layer immediately preceding the at least one / ayer][ e.g., the deactivation may be learned by training] [ e.g., the intermediate output segment is also called intermediate feature} / e.g., the intermediate output segment may be an output from the first layer or from any intermediate layer] [ e.g., in the case of the gating module being subdivided into multiple gating units, each gating unit may be configured to deactivate at least one deactivatable unit with which the gating unit is associated, based on the input segment and / or on at least one intermediate output segment outputted by a layer (e.g. immediately) preceding the layer with which the gating unit is associated] [ e.g., in some examples, it is possible to completely skip at least one layer (but, in some examples, not the totality of the layers of the NN)] , an output interface configured to provide an output derived from the last layer of the NN [e.g. the output may be an output sequential signal, such as in time domain (e.g. having a sequence of time domain samples) or in frequency domain (e.g., having a sequence of time-frequency bins)), or in position domain (such as text, which provides information on speech e.g. based on the position of a phoneme)] [e.g. the output may be a processed audio signal (e.g. evolving in time), or a processed video signal (e.g., evolving in time, e.g. as a sequence of video images), or more in general a processed media signal (e.g. having audio signals and video signals combined with each other] [ e.g., the processed sequential data may be a text which may be in the time domain or in the position domain]\the input sequential data may evolve, e.g. monodimensionally, along a ordered domain (e.g. time, space, etc.)] [the output may be e.g. rendered and / or encoded, in some examples] . In accordance to a concept, the apparatus may be such that the gating module is configured [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] [more in general for all groups of deactivatable units or for only one group of deactivatable unit, or for each group of deactivatable unit independently from the other groups of deactivatable units] to decide how many deactivatable units are deactivated [e.g. the number of deactivatable units to be deactivated may be an absolute number (such as a cardinal number of deactiva table units to be deactivated) or a ratio (e.g. operating as a percentage) of deactivatable units to be deactivated out of the total deactivatable units] [the gating module may output a number (either as an absolute number or relative number) of layers (or groups of deactivatable units) to be non-deactivated, and may be expressed as a utilization factor in the interval between 0 (or 0%) and 1 (or 100%), wherein a low value (e.g. close to 0) implies most of the deactivatable units of the at least one layer (or at least one group of deactivatable units) are to be deactivated, whale a value of I implies that all the deactivatable units of the at least one layer (or at least one group of deactivatable units) are activated and no deactivatable unit is deactivated at all, and a value close to I implies that almost all the deactivatable units are activated and only some deactivatable units are deactivated; in some examples, a special rule may be defined, that a utilization factor equal to 0 implies that the at least one layer (or at least one group of deactivatable units) is completely skipped; in some examples, a selection between a first number of deactivations and a second number of deactivations may be carried out, where the first number of deactivations (e.g. 1 or more than 1) in greater than the second number of deactivations (0 or more than 0, but less than the first number of deactivations)] [e.g., in case multiple layers (or groups of deactivatable units) are subjected to deactivation, in some examples for each layer (or group of deactivatable units) subjected to deactivation a layer-specific number of deactiva table units to be deactivated (or group-specific number of deactivatable units to be deactivated) may be decided by the gating module, and in alternative examples multiple layers or groups of deactivatable units (e.g. all the layers or groups of deactivatable units) may be subjected to the same number of deactivatable units to be deactivated (or, more in particular, to the same relative number (e.g. percentage) of deactivatable units to be deactivated)] based on the input segment and / or on at least one intermediate output segment [the decision on how many deactivatable units to deactivate may be learned by training; the decision may be based on an inference and / or a prediction] [e.g., in the case of multiple gating units associated with different layers or different groups of deactivatable units, each gating unit may be configured to decide, for the layers or groups of deactivatable units which it controls, independently of the other gating units].

[0155] The apparatus may be such that the gating module is configured, for at least one deactivatable unit [e.g. in some cases for al! the layers or groups of deactivatable units or for only one layer or only one group of deactivatable units, or for each layer or group of deactivatable units independently from the other ones] [more in general, for all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from the other ones], to decide how many deacti- vatable units to deactivate deterministically [e.g. based on the input segment and / or on at least one intermediate output segment] .

[0156] The apparatus may be such that the gating module is configured, for at least one deactivat- able unit [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] [more in general , for all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from, the other ones] , to decide how many deactivatable units to deactivate randomly [e.g. based on a random deactivation in such a way that the deactivated, deactivatable units are selected randomly]] e.g., in some cases, however, the random decision only interests some types of deactivatable units to be deactivated; for example, in the case in which the deactivatable units are both neurons and connections (e.g. each connection being associated with a particular weight), then the random deactivation may only interest, one particular type (e.g., only neurons but no connections, or only connections but no neuron, or both neurons and connections)] .

[0157] The apparatus may be such that the gating module is configured [e.g. for all the layers (or groups of deactivated)! e units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] to decide which deacti- vatable units to deactivate [e.g., the gating module may be configured to deactivate the at least one deactivatable unit based on the input segment or based on the intermediate output segment outputted by the layer (or group of deactivatable units) immediately preceding the at least one layer or group of deactivatable uni.ts][the deactivation may be learned by training; the decision may be based on an inference and / or a prediction fe.g., in the case of multiple gating units associated, with different layers (or more in general associated with different groups of deactivatable units), each gating unit may be configured to decide for the layers (or more in general for the groups of deactivatable units) which it controls independently of the other gating units].

[0158] The apparatus may be such that the gating module is configured [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] [more in general for all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from the other deactivatable units, or for all the groups of deactivatable units or for only one group of deactivatable units, or for each group of deactivatable units independently from the other groups of deactivatable units] to decide which deactivatable units to deactivate deterministi- cally ]e.g. based on the ranking] .

[0159] The apparatus may be such that the gating module is configured [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] [more in general [or all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from the other deactivatable units, or for all the groups of deactivatable units or for only one group of deactivatable units, or for each group of deactiva table units independently from the other groups of deactivatable units] to decide which deactivatable units to deactivate deterministi- cally based on a ranking on the deactivatable units.

[0160] The apparatus may be such that the gating module is configured [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from, the other ones] [more in general for all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from the other ones, or for all the groups of deactivatable units or for only one group of deactivatable units, or for each group of deactivatable units independently from the other groups of deactivatable units] to decide which deactivatable units being neurons to deactivate based on the weight values of the connections entering in, and / or exiting out, each deactivatable unit, so as to provide a lower ranking to the deactivatable units that have comparatively low weight values of the connections entering and / or exiting [e.g., in average, e.g. in norm, such as magnitude] than deacti- vatable units that have comparatively high weight values of the connections entering and / or exiting (e.g., in average, e.g. in norm, such as magnitude) [e.g. in case of deactivation of some deactivatable units (e.g. neurons, patterns etc.) in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those that have a low value (e.g. in norm, such as magnitude) because their scaling would imply a computational cost with low benefits; this is in particular important in the cases in which the deactivation is based on the input segment (see for example claims 26ff); e.g. in case of low noisiness it may be preferable to deactivate the deactivatable units which have entering and / or exiting connections having weight values (e.g., in average, e.g. in norm, such as magnitude) 0 or closer to 0, to reduce the computational complexity, while in case of higher noisiness it may be preferred to notwithstanding maintain also those deactivatable units, to increase the model capacity]-

[0161] The apparatus may be such that the gating module is configured [e.g. for all the layers (or groups o f deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independen tly from the other ones] [more in general for all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from the other ones, or for all the groups of deactivatable units or for only one group of deactivatable units, or for each group of deactivatable units independently from the other groups of deactivatable units] to decide which deactivatable units to deact ivate based on their importance quan- tified by a score function of at least the weight values of the connections entering in and / or exiting out the deactivatable units being neurons [and / or based on a score function computed from the weights values (e.g., norm, correlation) and / or the inputs (e.g., to include activation)] [e.g. in case of deactivation of some deactivatable units in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those that have a low importance because their scaling would imply a computational cost with low benefits; this is in particular important in the cases in which the deactivation is based on the input segment (see for example claims 26ff.); e.g. in case of non-important deactivatable unit, it may be preferable to avoid it, while in case of increased importance of the deactivatable unit, it may be preferable to notwithstanding maintain activated also those deactivatable units, to increase the quality].

[0162] The apparatus may be such that the gating module is configured [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] [more in general for all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from the other ones, or for all the groups of deactivatable units or for only one group of deactivatable units, or for each group of deactivatable units independently from the other groups of deactivatable units] to decide which deactivatable units being neurons or patterns to deactivate based on the weighted values inputted to the deactivatable units or outputted by the deactivatable units, so as to provide a lower ranking to the deactivatable units that have comparatively low weighted in- putted values or lower weighted outputted values [e.g., in average, e.g. in norm, such as magnitude] than deactivatable units that have comparatively high weight values entering have comparatively high weighted inputted values or high weighted outputted values [e.g., in average, e.g. in norm, such as magnitude] [e.g. in case of deactivation of some deactivatable units (e.g. neurons, patterns etc.) in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those deactivatable units that output (or that receive in input) a low value (e.g. in norm, such as magnitude) because their scaling would imply a computational cost with low benefits; this is in particular important in the cases in which the deactivation is based on the input segment (see for example claims 26fff[.

[0163] The apparatus may be such that the gating module is configured [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] [more in general for all the deactivatable units or for only one deactivatable unit, or for each deactivatable unit independently from the other ones, or for all the groups of deactivatable units or for only one group of deactivatable units, or for each group of deactivatable units independently from the other groups of deactivatable units] to decide which deactivatable units to deactivate, the deactivatable units being connections between neurons or patterns and having weights, so as to deactivate the deactivatable units based on the weight values of the deact ivatable units being connect ions, so as to provide a lower ranking to the deactivatable units being connections that have comparatively low weight values [e.g., in average, e.g. in norm, such as magnitude] than deactivatable units being connections that have comparatively high weight values (e.g., in average, e.g. in norm, such as magnitude) [e.g. in case of deactivation of some deactivatable units being connections in at least one layer (or group of deactivatable units) it may be chosen to first deactivate those that have a low weight value (e.g. in norm, such as magnitude) because their scaling would imply a computational cost with low benefits; e.g. in case of low noisiness it may be preferable to deactivate the deactivatable units being connection having weight values (e.g., in average, e.g. in norm, such as magnitude) 0 or closer to 0 than those having higher value, to reduce the computational complexity, while in case of higher noisiness it may be preferred to notwithstanding maintain also those deactivatable units (e.g. lower ranked deactivatable units), to increase the model capacity].

[0164] The apparatus may be such that the gating module is configured, for at least one group of deactivatable units [e.g. for all the layers (or groups of deactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable units) independently from the other ones] , to decide which deactivatable units to deactivate randomly [e.g. based on a random deactivation in such a way that the deactivated deactivatable units are selected randomly] [e.g., in some cases, however, the random decision only interests some types of deactivatable units to be deactivated; for example, in the case in which the deactivatable units are both neurons and connections (e.g. each connection being associated with a particular weight), then the random deactivation may only interest one particular type (e.g., only neurons but no connections, or only connections but no neuron, or both neurons and connections)] .

[0165] The apparatus may be such that the gating module is configured, for at least one group of deactivatable units [e.g. for all the layers (or groups ofdeactivatable units) or for only one layer (or group of deactivatable units), or for each layer (or group of deactivatable unitsfindependently from the other ones], to apply a utilization factor [e.g. indicated with if e.g. in formulae (2), (3), and (4), and indicated as a scalar u, in formula ([)], wherein the deactivatable units of at least one group of deactivatable units are ranked from a first-ranked deactivatable unit to a last-ranked deactivatable unit, wherein a comparatively high utilization factor implies a comparatively high number of acti- vated deactivatable units including lower ranked deactivatable units, and a comparatively low utili- zation factor implies a comparatively low number of activated deact ivatable units, wherein the deac- tivatable units are activated, in accordance to their decreasing ranking, at the increase of the ut iliza- tion factor [the least ranked deactivatable unit(s) may therefore be the least important, and the highest ranked deactivatable unit unit(s) may therefore be the most important] [it may happen that, in some examples, some extremely important elements of the NN (neurons, patterns, connections (carrying weights), bias(es), input elements of the input segment, learnable parameters of the NN, etc.) are not deactivatable, in view of their importance; for example, in one single layer of a fully connected NN, there can be some deactivatable neurons, but also some non-deactivatable neurons] [The least ranked deactivatable unit(s) may be deactivated when the utilization factor is highest, and the highest ranked deactivatable unit(s) may be deactivated only when the utilization factor is minimum ranked activated deactivatable units, and the lower the utilization factor, the higher the ranking of the lowest-ranked activated deactivatable units] [the lower the ranking o f a deactivatable unit, the higher the utilization factor shall be to cause its activation, and the higher the ranking of a deactivatable unit, the lower the utilization factor can be to cause its activation] [e.g., in the case of multiple gating units associated with different layers (or groups of deactivatable layers), each gating unit may be configured to apply a utilization factor independently of the other gating units.

[0166] The apparatus may be such that the ranking is predefined [e.g. before training] [e.g., in some examples, when the training start the weights are already ordered and the ranking doesn ’t change therefore during inference the ranking is predefined] .

[0167] The apparatus may be such that the ranking of the deactivatable units being neurons is at least partially conditioned by weight values of the connections entering in and / or exiting out the deactivatable units of at least one group of deactivatable units.

[0168] The apparatus may be such that the ranking is derived by reordering the deactivatable units based on a score function of the weight values of the connections entering in and / or exiting out the deactivatable units, the deactivatable units being neurons, [the score [unction may be or involve for example the magnitude of the weight values]

[0169] The apparatus may be such that the gating module is configured to apply one single utiliza- tion factor to all, or at least to a plurality of, deactivatable units of the NN. [e.g. formula (1), Fig. 2] [e.g., in some examples, multiple layers have different numbers of units (e.g. neurons), but if the utilization factor is applied to all of them as a relative number (e.g. a percentage), then all of the layers deactivate a number of deactivatable layers proportional to the utilization factor)]

[0170] The apparatus may be such that the deactivatable units being neurons or patterns and each deactivatable unit being in a specific layer, wherein the gating module is configured to apply, based on the input segment [e.g. based uniquely of the input segment] , a layer-specific utilization factor to each layer [e.g. formula (2)] [e.g., in this way different layers have in genera! different utilization factors] or to apply a layer-specific utilization factor to each group of layers [e.g., in this way multiple layers may have the same utilization factor, but in general they differ from the utilization factors for other layers or groups of layers] [this may be the case of multiple gating units associated with different layers, e.g. each gating being independent of the other gating units].

[0171] The apparatus may be such that at least one group of deactivatable units is formed by deac- tivatable units controlled with the same utilization factor [e.g. by different gating units within the same gating module] , each group of deactivatable units being formed by deactivatable units in a same layer(s), but different groups have no deactivatable units in the same layer(s). [e.g. each single gating unit in the gating module may control one unique "layer” or a plurality of unique "layers ”, so that two gating units do not control the same "layer”; this may imply that all the layers controlled by the same gating unit apply the same utilization factor, while the layers controlled by different in this case, each single gating unit in the gating module may control one unique "layer” (i.e., the totality of deactivatable connections entering in the same neuronal layer) or a plurality of unique "layers ” (i.e., the totality of deactivatable connections entering in same neuronal layers), so that two gating units do not control the same "layer(s) ” (i.e. two gating units do not control the same groups of deactivatable connections entering in the same neuronal layer(s)); this may imply that all the connections controlled, by the same gating unit apply the same utilization factor, while the connections controlled by different gating units in principle apply different utilization factors]

[0172] The apparatus may be such that the deactivatable units being connections between neurons, and each deactivatable unit being associated with a group of deactivatable units entering in one par- ticular layer or exiting from one particular layer, wherein the gating module is configured to apply, based on the input segment [e.g. based uniquely of the input segment], a layer-specific utilization factor to each layer [e.g., in this way different groups of units have in general different utilization factors] or to apply a group-specific utilization factor to each group of deactivatable units [e.g., in this way multiple groups of units may have the same utilization factor, but in general they differ from the utilization factors for other groups of units] [this may be the case of multiple gating units associated with different layers, e.g. each gating being independent of the other gating units}.

[0173] The apparatus may be such that the gating module is configured to apply, to at least one deactivatable unit [e.g. to all the deactivatable units of the NN] or a group of deactivatable units, a unit-specific or group-specific utilization factor based at least on an intermediate output segment [e.g., intermediate feature] outputted by at least one of the preceding deactivatable units [e.g. by the immediately preceding deactivatable unit and / or by a series of immediately preceding deactivatable units].

[0174] The apparatus may be such that the gating module is configured to apply, to at least one deactivatable unit [e.g. to all the layers of the NN] or a group of deactivatable units, a group-specific utilization factor based both on at least one intermediate output segment [e.g., intermediate feature] outputted by at least one of the preceding layers and on the input segment [this may be the case of multiple gating units associated with different layers, e.g. each gating being independent of the other gating units} .

[0175] The apparatus may be such that the input segment is in a time-based sequence [e.g. an input sequential signal, such as in time domain (e.g. having a sequence of time domain samples) or in frequency domain (e.g., having a time-based sequence of timefrequency bins)] [e.g. an audio signal (e.g. evolving in time), on a video signal (e.g., evolving in time, e.g. as a sequence of video images), or more in general a media signal (e.g. having audio signals and video signals combined with each other] [e.g., in some examples, the input segment may have a time-based sequence of frames, each [ram e h a ving m u Itiple freq u e n cy b ins]

[0176] The apparatus may be such that the input segment is in a position-based sequence [e.g., such as text, which provides information on speech e.g. based on the position of a phoneme] [in some examples, the sequential data may be a text in the time domain or in the position domainfie.g., the input sequential data may evolve, e.g. monodimensionally, along an ordered domain (e.g. space, etc.)}

[0177] The apparatus may be such that the gating module is configured to select between deactivat- ing a second number of deactivatable units and deactivating a first number of deactivatable unit which is 0 or greater than 0, wherein the second number of deactivatable units is greater than the first number of deactivatable units.

[0178] The apparatus may be such that the gating module is configured to classify the input segment, based at least on the input segment [sec for example Figs. 1, 2, and 3], between at least a first clas- sification result, indicative of the input segment requiring a processing at a first computational com- plexity, and a second classification result, indicative of the input segment requiring a processing at a second computational complexity which is higher than the first computational complexity [e.g. the second (higher) computational complexity may be necessary for highly noisy signals (e.g. low signal- to-noise ratio, e.g. noisy speech in case of the input segment being representative of an audio signal), and the first (lower) computational complexity may be used for lowly noisy signals (e.g. high signal- to-noise ratio, e.g. clean speech in the case of the input segment being representative of the audio signal)], wherein the gating module is configured to deactivate a greater number of deactivatable units [e.g. for one or more layers (or groups of deactivatable units), e.g. for all the layers (or groups of deactivatable units)] in the case of the first classification result than in the case of the second classification result [e.g. this may be valid for one or more layers (or groups of deactivatable units), e.g.. for all the layers (or groups of deactivatable units)] [e.g., in some examples it is admitted that in the case of the second classification result no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the second, classification result].

[0179] The apparatus may be such that the gating module is configured to classify the input segment, based at least on the input segment [see for example Figs. I, 2, and 3], between a plurality [e.g. 2 or more than 2] of classification results indicative of different computational complexities, wherein the gating module is configured to deactivate a greater number of deactivatable units for input segment(s) having classification result(s) indicat ive of the input segment requiring a processing at comparatively low computational complexity than for input segment(s) having classification result(s) indicative of the input segment requiring a processing at comparatively high computational complexity [e.g., in some examples it is admitted that in the case of the classification result being indicative of the input segment requiring highest computational complexity, no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the classification result being indicative of the input segment requiring highest computational complexity; in addition or in alter native, it may be admitted that in the case of the classification result being indicative of the input segment requiring lowest computational complexity, all the deactivatable units are deactivated, which may cause, in some examples, that the output result of the NN is the same of the input segment of the NN in the case of the classification result being indicative of the input segment requiring highest computational complexity] .

[0180] The apparatus may be such that the gating module is configured to classify the input segment, based at least on the output result [see. for example Fig. 5B] [the output result may be, for example, the output result of a previous time instant or previous position] , between at least a first classification result, indicative of the input segment requiring a processing at a first computational complexity, and a second classification result, indicative of the input segment requiring a processing at a second com- putational complexity which is higher than the first computational complexity [e.g. the second (higher) computational complexity may be necessary for highly noisy signals (e.g. low signal-to- noise ratio, e.g. noisy speech in case of the input segment being representative of an audio signal), and the first (lower) computational complexity may be used for lowly noisy signals (e.g. high signal- to-noise ratio, e.g. clean speech in the case of the input segment being representative of the audio signal)] , wherein the gating module is configured to deactivate a greater number of deactivatable units for at least one group of deactivatable units [e.g. for one or more layers (or groups of deactivatable units), e.g. for all the layers (or groups of deactivatable units)] in the case of the first classi- fication result than in the case of the second classification result [e.g. this may be valid for one or more layers (or groups of deactivatable units), e.g.. for all the layers (or groups of deactivatable units)] [e.g., in some examples it is admitted that in the case of the second classification result no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the second classification result].

[0181] The apparatus may be such that the gating module is configured to classify the input segment, based at least on the output result [e.g., see for example Fig. 5B][ the output result may be, for example, the output result of a previous time instant or previous position] , between a plurality (e.g. 2 or more than 2] of classification results indicative of different computational complexities, wherein the gating module is configured to deactivate a greater number of deactivatable units for input seg- ment(s) having classification result(s) indicative of comparatively low computational complexity than for input segment(s) having classification result(s) indicative of comparatively high computa- tional complexity (e.g., in some examples it is admitted that in the case of the classification result being indicative of the input segment requiring highest computational complexity, no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated, deactivatable units is 0 in the case of the classification result being indicative of the input segment requiring highest computational complexity; in addition or in alternative, it may be admitted that in the case of the classification result being indicative of the input segment requiring lo west compu tational complexity, all the deactivatable units are deactivated, which may cause, in some examples, that the output result of the NN is the same of the input segment of the NN in the case of the classification result being indicative of the input segment requiring highest computationcd complexity] .

[0182] The apparatus may be such that the gating module is configured to classify the input segment, based at least on an intermediate output result [e.g. an output from the first layer or an output from any hidden layer] [e.g., see for example Fig. 3] [e.g., the intermediate output result may be relative to the same time instant or position of the input segment] , between at least a first classification result, indicative of the input segment requiring a processing at a first computational complexity, and a second classification result, indicative of the input segment requiring a processing at a second com- putational complexity which is higher than the first computational complexity [e.g. the second (higher) computational complexity may be necessary for highly noisy signals (e.g. low signal-to- noise ratio, e.g. noisy speech in case of the input segment being representative of an audio signal), and the first (lower) computational complexity may be used for lowly noisy signals (e.g. high signal- to-noise ratio, e.g. clean speech in the case of the input segment being representative of the audio signal)], wherein the gating module is configured to deactivate a greater number of deactivatable units for at least one group of deactivatable units [e.g. for one or more layers (or groups of deactivatable units), e.g. for all the layers (or groups of deactivatable units)] in the case of the first classi- fication result than in the case of the second classification result [e.g. this may be valid for one or more layers (or groups of deactivatable units), e.g.. for all the layers (or groups of deactivatable units)] [e.g., in some examples it is admitted that in the case of the second classification result no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the second classification result] [when the gating module classifies the input segment based at least on an intermediate output result, this may in particular be used, in the case in which multiple gating units apply different utilization factors to different layers (or anyway different groups of deactivatable units; for example, each layer may be controlled by a layer- specific utilization factor applied by a gating unit which has, in input, the intermediate output result which immediately precedes the layer (and, in some cases, also the input segment)].

[0183] The apparatus may be such that the gating module is configured to classify the input segment, based at least on an intermediate output result [e.g. an output from the first layer or an output from any hidden layer] [e.g., see for example Fig. 3f[ e.g., the intermediate output result may be relative to the same time instant or position of the input segment] , between a plurality ]e.g. 2 or more than 2] of classification results indicative of different computational complexities, wherein the gating module is configured to deactivate a greater number of deactivatable units for an input segment hav- ing a classification result indicative of comparatively low computational complexity than for an input segment having a classification result indicative of comparatively high computational complexity [e.g., in some examples it is admitted that in the case of the classification result being indicative of the input segment requiring highest computational complexity, no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the classification result being indicative of the input segment requiring highest computational complexity; in addition or in alternative, it may be admitted that in the case of the classification result being indicative of the input segment requiring lowest computational complexity, all the deactivatable units are deactivated, which may cause, in some examples, that the output result of the NN is the same of the input segment of the NN in the case of the classification result being indicative of the input segment requiring highest computational complexity] [when the gating module classifies the input segment based at least on an intermediate output result, this may in particular be used in the case in which multiple gating units apply different utilization factors to different layers (or anyway different groups of deactivatable units; for example, each layer may be controlled by a layer-specific utilizationfactor applied by a gating unit which has, in input, the intermediate output result, which immediately precedes the layer (and, in some cases, also the input, segment)] .

[0184] The apparatus may be such that the gating module is configured to classify the input segment based at least on a noisiness measurement of the input segment, or a processed version thereof, be- tween at least the first classification result, indicative of comparatively low noisiness [e.g. clean speech], and the second classification result, indicative of comparatively high noisiness [e.g. noisy speech], so that the gating module deactivates a greater number of deactivatable units for at least one group of deactivatable units in the case of the comparatively low noisiness than in the case of the comparatively high noisiness [e.g., in some examples it is admitted that in the case of the second classification result no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the second classification result].

[0185] The apparatus may be such that the gating module is configured to classify the input segment based at least on a noisiness measurement of the input segment, or a processed version thereof, be- tween a plurality of classification results indicative of different noisinesses, wherein the gating mod- ule is configured to deactivate a greater number of deactivatable units for input segment(s) having classification result(s) indicative of low noisiness than for input segment(s) having classification re- sults) indicative of comparatively high noisiness [e.g., in some examples it is admitted that in the case of the classification result being indicative of the input segment requiring highest computational complexity, no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the classification result being indicative of the input segment requiring highest computational complexity; in addition or in alternative, it may be admitted that in the case of the classification result being indicative of the. input segment requiring lo west computational complexity, all the deactivatable units are deactivated, which may cause, in some examples, that the output result of the NN is the same of the input segment of die NN in the case of the classification result being indicative of the input segment requiring highest computational complexity] .

[0186] The apparatus may be such that the gating module is configured to classify the input segment based on a harmonieity measurement of the input segment, or a processed version t hereof, between at least the first classification result, indicative of comparatively high harmonieity [e.g. clean speech] , and the second classification result, indicative of comparatively low harmonieity [e.g. noisy speech] , so that the gating module deactivates a greater number of deactivatable units for at least one group of deactivatable units in the case of comparatively high harmonieity than in the case of the comparatively low harmonieity [e.g., in some examples it is admitted that in the case of the second classification result no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the second classification result] .

[0187] The apparatus may be such that the gating module is configured to classify the input segment based at least on a harmonieity measurement of the input segment, or a processed version thereof, between a plurality of classification results indicative of different harmonicities, wherein the gating module is configured to deactivate greater number(s) of deactivatable units for input segment(s) hav- ing classification result(s) indicative of comparatively high harmonieity than for input segment(s) having classification result(s) indicative of comparatively low harmonieity [e.g., in some examples it is admitted that in the case of the classification result being indicative of the input segment requiring highest computational complexity, no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the classification result being indicative of the. irnput segment requiring highest computational complexity; in addition or in alternative, it may be admitted that in the case of the classification result being indicative of the input segment requiring lowest computational complexity, all the deactivatable units are deactivated, which may cause, in some examples, that the output result of the NN is the same of the input segment of the NN in the case of the classification result, being indicative of the input segment requiring highest, computational complexity] .

[0188] The apparatus may be such that the gating module is configured to classify the input segment based on an interference measurement between at least the first classification result, indicative of comparatively low interference, and the second classification result, indicative of comparatively high interference, so that the gating module deactivates a greater number of deactivatable units for at least one group of deactivatable units in the case of the comparatively low interference than in the case of the comparatively high interference [e.g., in the case of speech separation, the number of speakers [or, more in general, audio sources] in the input segment might correspond to the level of interference. [in some examples it is admitted that in the case of the second classification result no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the second classification result].

[0189] The apparatus may be such that the gating module is configured to classify the input segment based at least on an interference measurement of the input segment, or a processed version thereof, between a plurality of classification results indicative of different levels of interference, wherein the gating module is configured to deactivate a greater number of deactivatable units for an input seg- ment having a classification result indicative of comparatively low interference than for an input segment having a classification result indicative of comparatively high interference [e.g., in the case of speech separation, the number of speakers in the input segment might correspond to the level of interference. Speaker counting can be also achieved using neural networks] [e.g., in some examples it is admitted that in the case of the classification result being indicative of the input segment requiring highest computational complexity, no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the classification result being indicative of the input segment requiring highest computational complexity; in addition or in alternative, it may be admitted that in the case of the classification result being indicative of the input segment requiring lowest computational complexity, all the deactivatable units are deactivated, which may cause, in some examples, that the output result of the NN is the same of the input segment of the NN in the case of the classification result being indicative of the input segment requiring highest computational complexity] .

[0190] The apparatus may be such that the gating module is configured to classify the input segment, based at least on an additional [e.g. auxiliary] signal fe.g., see for example Fig. SBffe.g., the additional signal may be indicative of the context] , between at least a first classification result, indicative of the input segment requiring a processing at a first computational complexity, and a second classi- fication result, indicative of the input segment requiring a processing at a second computational com- plexity which is higher than the first computational complexity [e.g. the second (higher) computational complexity may be necessary in case the additional input segment is indicative of a context that indicates the necessity of higher complexity; for example, the input segment can be an audio signal and the NN can be used to separate a speech component from a noise component, and the second input segment may be a video signal from which the gating module shall classify the input segment between speech and non-speech by analysing the video signal, e.g. by recognizing the lip mo vement; the other input may be, for example, another input segment with the same voice of the speaker, so that it is possible to classify the input segment as being in the second classification result in the case of the input segment differing from the additional audio signal for less than a predetermined threshold, and to classify the input segment as being in the first classification result in the case of the input segment differing from the additional audio signal for more than the predetermined threshold] , wherein the gating module is configured to deactivate a greater number of deactivatable units for at least one group of deactivatable units in the case of the first classification result than in the case of the second classification result [e.g., in some examples it is admitted that in the case of the second classification result no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the second classification result].

[0191] The apparatus may be such that the gating module is configured to classify the input segment, based at least on an additional [e.g. auxiliary] signal [see for example Fig. 5B][e.g., the additional signal may be indicative of the context] , between a plurality of classification results, indicative of the input segment requiring processings at different computational complexities [e.g. higher computational complexities may be necessary in case the additional input segment is indicative of a context that indicates the necessity of higher complexity, and. lower computational complexities may be necessary in case the additional input segment is indicative of a context that indicates the necessity of lo wer complexity; for example, the input segment can be an audio signal and the NN can be used to separate a speech component from a noise component, and the second input segment may be a video signal from which the gating module shall classify the input segment between speech and non-speech by analysing the video signal, e.g. by recognizing the lip movement; the other input may be, for example, another input segment with the same voice of the speaker, so that it is possible to classify the input segment as being in the second classification result in the case of the input segment differing from the additional audio signal for less than a predetermined threshold, and to classify the input segment as being in the first classification result in the case of the input segment differing from the additional audio signal for more than the predetermined threshold] , wherein the gating module is configured so that classification results indicative of the input segment requiring processings at com- paratively reduced computational complexities imply a deactivation of a comparatively increased number of units, and classification results indicative of the input segment requiring processings at comparatively increased computational complexities imply a deactivation of a comparatively re- duced number of units, or no deactivation at all [e.g., in some examples it is admitted that in the case of the classification result being indicative of the input segment requiring highest computational complexity, no deactivatable unit is deactivated at all, which therefore cause that the number of deactivated deactivatable units is 0 in the case of the classification result being indicative of the input segment requiring highest computational complexity; in addition or in alternative, it may be admitted that in the case of the classification result being indicative of the input segment requiring lo west computational complexity, all the deactivatable units are deactivated, which may cause, in some examples, that the output result of the NN is the same of the input segment of the NN in the case of the classification result being indicative of the input segment requiring highest computational complexity]. The apparatus may be such that the input segment is received from a bitstream, and the ad- ditional signal is obtained from signalling in the bitstream.

[0192] The apparatus may be such that it is configured to skip at least one layer [e.g. by deactivating all the deactivatable units of the layer, which is in this case, as special rule, are bypassed] by deac- tivating all the deactivatable units of the layer, and bypassing the at least one layer.

[0193] The apparatus may be such that it is configured to classify the inputs segment between more than two classification results indicative of different levels of computational complexity required, the gating module being configured to deactivate, for at least one group of deactivatable units, the deactivatable units in accordance with the classified level of computational complexity required, so that a higher computational complexity required implies a lower number of deactivation of deacti- vatable units for the at least one group of deactivatable units, and a lower computational complexity required implies a higher number of deactivation of deactivatable units for the at least one group of deactivatable units.

[0194] The apparatus may be such that the NN is inputted with the additional [e.g. auxiliary] signal in at least one of the layers, [e.g., For example, in a target speaker extraction application, target speaker information (e.g., speech snippet from the target speaker, visual frames of Up movements) can be provided to inform the NN about the identity of the speaker of interest.]

[0195] The apparatus may be such that the gating module includes a second NN, or another learning unit, to classify the input segment. The apparatus may be such that the gating module includes a deterministic classifier unit to classify the input segment.

[0196] The apparatus may be such that the NN is trained to reduce noise from the input segment being indicative of an audio signal, so as to provide the output result as a denoised version of the input segment. The apparatus may be such that the NN is trained to extract a target component from the input segment being indicative of an audio signal, so as to provide the output result as a target com- ponent of the input segment.

[0197] The apparatus may be such that the NN is trained to extract a speech component from the input segment being indicative of an audio signal, so as to provide the output result as a speech component of the input segment .

[0198] The apparatus may be such that the NN is trained to extract a component from one specific audio source [e.g. a specific speaker person] from the input segment being indicative of an audio signal with multiple audio sources [e.g. multiple speaker persons] , so as to provide the output result as a component of the audio signal from the specific audio source [e.g. specific person] . The apparatus may be such that the NN is trained to provide the output result as having a higher dimensionality than the dimensionality of the input segment . The apparatus may be such that the deactivatable units include neurons.

[0199] The apparatus may be such that the deactivatable units include patterns, the NN being a convolutional NN.

[0200] The apparatus may be such that the deactivatable units include bias.

[0201] The apparatus may be such that the deactivatable units include connections carrying weights.

[0202] The apparatus may be such that the deactivatable units include input elements of the input segment [e.g. positions in the array of the input segment] .

[0203] The apparatus may be such that the deactivatable units include learnable parameters of the NN.

[0204] The apparatus may be such that it is configured for performing audio filtering [e.g. TF- masking for speech enhancement / separation] , the input segment representing an audio signal [e.g. a portion of the audio signal], wherein the NN is configured to produce at least one output segment by processing [e.g. masking] the input signal.

[0205] The apparatus may be such that it is configured to choose among operating at a first mode and operating at a second mode, wherein the first mode implies controlling the deactivation of the deactivatable units according to a first criterion, and the second mode implies controlling the deacti- vation of the deactivatable units according to a second criterion [the criterion may involve at least one of: the number of deactivatable units; the ranking of the deactivatable units; which deactivatable units; the type (neurons, patterns, connections (having weights), input elements, learnable parameters of the NN, biases, etc.); in which layer the deactivatable units are; which utilization factor to apply; which classification, etc.]

[0206] The apparatus may be such that the choice between the first mode and the second mode is performed deterministically [e.g. based on a coarse classification]

[0207] In accordance to a concept, there is provided a method [e.g. a signal processing method; the method may be for speech enhancement (e.g. noise reduction, target speaker extraction or more in general target component extraction e.g. speech separation; the method may be for media enhancement and / or video enhancement] comprising: receiving an input segment [e.g., 1J of an input sequential data [e.g. an input sequential signal, such as in time domain (e.g. having a sequence of time domain samples) or in frequency domain (e.g., having a time-based sequence of time-frequency bins)), or in position domain (such as text, which provides information on speech e.g. based on the position of a phoneme)] [e.g. an audio signal (e.g. evolving in time), or a video signal (e.g., evolving in time, e.g. as a sequence of video images), or more in general a media signal (e.g. having audio signals and video signals combined with each other] [e.g., in some examples, the sequential data may be a text in the time domain or in the position domainfithe input sequential data may evolve, e.g. monodimerisionally, along an ordered domain (e.g. time, space, etc.)] through a neural network, NN, deriving an output result [e.g a processed version of the input segment (e.g. a voice component (or another target component) of the input segment in the case the input signal being an audio signal being a mixture of speech and noise), and / or a noise-reduced version an audio input signal, and / or a speaker component (or more in general target component) of an audio input signal; or a compressed (e.g. encoded) or decompressed (e.g. decoded) version of the input segment or a classification result on the input segment] by processing the input segment [IJ through the NN [e.g. a slimmable NN] having a number of layers [e.g., at least 2 layers] from a first layer [e.g, layer 1, e.g. as in Figs. 2 and 5 J to a last layer [e.g. layer M, e.g. as in Figs. 2 and 3] [e.g. with the possible interposition of at least one hidden layer between the first layer and the last layer], the NN having a predetermined number of deactivatable units [e.g, each layer may have a specific number of deactivatable units and the specific number of deactivatable units may be the same or different from a predetermined number of deactivatable units of at least one other layer] [in some examples (e.g. in a fully-connected NN), each deactivatable unit may be a neuron, and each neuron of each layer (apart from the last layer) may be connected with each neuron of the immediately preceding layer through a learnable neuron-to-neuron specific weight, and each layer (apart from the first layer) may be connected with each neuron of the immediately preceding layer through a learnable neuron-to-neuron specific weight, e.g. in the case of a fully connected NN][in some examples (e.g. in a convolutional NN, CNN), each deactivatable unit may be a specific predetermined pattern (filter) to be convoluted along the input segment (the input segment may be, in this case, a tensor (e.g. a multi-dimensional tensor, such as a 2-dimensional tensor or 3-dimensional tensor or having more than 3 dimensions), to provide a tensor result (e.g. the tensor result being obtained by multiplying the values of the pattern by the values of the input segment))] [e.g, in some examples, for each layer the deactivatable units are ranked from a first-ranked deactivatable unit to a last- ranked deactivatable unit][e.g., the output of the NN may be, in some examples, sequential, e.g. there may be one output value [e.g., scalar or tensorial (e.g. vectorial, matricial, 3-D-tensorial, etc.) for each value of the ordered domain (e.g. time, space, etc.); in other examples, the output value may be an integral feature of the input segment (e.g. a value that provides information on multiple values of the ordered domain, e.g. a value that provides information on the entire input segment, such as a time period in the case the ordered domain is time, or a spatial length (or positional length) in the case the ordered domain is a space (or a position))]]; deactivating at least one deactivatable unit [e.g. a number of deactivatable units between 1 and all the deactivatable units (in some examples, it may be between 1 deactivatable unit and all the deactivatable units)] based on the input segment and / or on at least one intermediate output segment [e.g an output from the first layer or an output from any hidden layerfr.g., the gating may deactivate the at least one deactivatable unit of the at least one layer based on the input segment or based on the intermediate output segment outputted by the layer immediately preceding the at least one / ayer][e.g., the deactivation may be learned by training] [the intermediate output segment is also called intermediate frature][e.g., the intermediate output segment may be an output from the first layer or from any intermediate layer] [e.g., in the case of the gating being subdivided into multiple gating steps, each gating unit may be configured to configured to deactivate at least one deactivatable unit of the layer (or group of layers) with which the gating unit is associated, based on the input segment and / or on the on at least one intermediate output segment outputted by a layer (e.g. immediately) preceding the layer with which the gating unit is associated] [e.g., in some examples, it is possible to completely skip at least one layer (but not the totality of the layers of the NN)], outputting an output result derived from the last layer of the NN [e.g. the output may be an output sequential signal, such as in time domain (e.g. having a sequence of time domain samples) or in frequency domain (e.g., having a sequence of time-frequency bins)), or in position domain (such as text, which provides information on speech e.g. based on the position of a phoneme)] [e.g. the output may be a processed audio signal (e.g. evolving in time), or a processed video signal (e.g., evolving in time, e.g. as a sequence of video images), or more in general a processed media signal (e.g. having audio signals and video signals combined with each other] [e.g., the processed sequential data may be a text which may be in the time domain or in the position domain][e.g., the input sequential data may evolve, e.g. monodimensionally, along a ordered domain (e.g. time, space, etc.)] [e.g., the output may be e.g. rendered and / or encoded in some examples].

[0208] There is provided a non-transitory storage unit storing instructions which, when executed by a processor, cause the processor to perform the above mentioned methods.

[0209] Further characterization of the figures

[0210] Here below there are some characterizations of some figures in some examples:

[0211] Figure 1 : A high-level block diagram of the proposed method. The proposed method pro- cesses a segment of the input sequence, denoted by lt, and produces the output 0t. A slimmable neural network (SNN) comprising slimmable layers can be executed at different widths (e.g., number of units) for a given utilization factor(s), which is predicted from a gating mechanism by analyzing the characterist ics of the input segment and / or some intermediate features from the SNN.

[0212] Figure 2: Proposed method utilizing a global gating mechanism, where a single gate controls the width of all sliminable layers.

[0213] Figure 3: Proposed method utilizing a local gating mechanism. An independent gate for each slimmable layer in the SNN controlling its width by analyzing the intermediate features ftl. Option- ally, the input segment Itcan be provided to the gates as an additional input.

[0214] References [1] Y. Luo, Z. Chen, and I . Yoshioka, “Dual-path RNN: Efficient long sequence modeling for time- domain single-channel speech separation, ”m Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), May 2020, pp. 46-50.

[0215] [2] J. Chen, Q. Mao, and D. Liu, “Dual-path transformer network: Direct context-aware modeling for end-to-end monaural speech separation,” in Proc. Interspeech Conf., Oct. 2020, pp. 2642-2646.

[0216] [3] N. Zeghidour and D. Grangier, “Wavesplit: End-to-end speech separation by speaker clustering,” IEEE / ACM Trans. Audio, Speech, Lang. Process., vol. 29, pp. 2840-2849, Jul. 2021.

[0217] [4] C. Subakan, M. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong, “Attention is all you need in speech separation,” in Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), Jun. 2021 , pp. 21-25.

[0218] [5] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “Tf-gridnet: Integrating full- and sub-band modeling for speech separation,” IEEE / ACM Trans. Audio, Speech, Lang. Pro- cess., vol. 31 , pp. 3221-3236, Aug. 2023.

[0219] [6] E. Tzinis, Z. Wang, X. Jiang, and P. Smaragdis, “Compute and memory efficient universal sound source separation,” Journal of Signal Processing Systems, vol. 94, no. 2, pp. 245-259, Jul. 2022.

[0220] [7] Y. Bengio, N. L'eonard, and A. Courville, “Estimating or propagating gradients through stochas- tic neurons for conditional computation,” arXiv preprint arXiv: 1308.3432, 2013.

[0221] [8] Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang, “Dynamic neural networks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 7436-7456, Nov. 2022.

[0222] [9] EL Liu, S. Parajuli, J. Hostetler, S. Chai, and B. Bhanu, “Dynamically throttleable neural net- works,” Machine Vision and Applications, vol. 33, no. 4, p. 59, Jul. 2022.

[0223]

[0010] Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L. S. Davis, K. Grauman, and R. Feris, “Blockdrop: Dynamic inference paths in residual networks,” in Proc. IEEE Conf, on Computer Vision and Pattern Recognition (CVPR), June 2018.

[0224]

[0011] W. Xia, FI. Yin, X. Dai, and N. K. Jha, “Fully dynamic inference with deep neural networks,” IEEE Transactions on Emerging Topics in Computing, vol. 10, no. 2, pp. 962-972, Apr. 2022.

[0225]

[0012] T. Bolukbasi, J. Wang, 0. Dekel, and V. Saligrama, “Adaptive neural networks for efficient inference,” in Proc. Inti. Conf, on Machine Learning (ICML), vol. 70, Aug. 2017, pp. 527-536.

[0226]

[0013] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural Computation, Mar. 1991.

[0227]

[0014] Z. You, S. Feng, D. Su, and D. Yu, “SpeechMoE: Scaling to large acoustic models with dynamic routing mixture of experts,” in Proc. Interspeech Conf., Aug. 2021 , pp. 2077- > 2081.

[0228]

[0015] C. Li, G. Wang, B. Wang, X. Liang, Z. Li, and X. Chang, “Dynamic slimmable network,” in Proc. IEEE / CVF Conf, on Computer Vision and Pattern Recognition (CVPR), Jun. 2021 , pp. 8607- 8617.

[0016] > . , “Ds-neH H Dynamic weight slicing for efficient inference in cnns and vision transform- ers,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4430-4446, Apr. 2023.

[0229]

[0017] W. Eedus, B. Zoph, and N. Shazecr, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” The Journal of Machine Learning Research, vol. 23, no. 1 , pp. 5232-5270, Apr. 2022.

[0230]

[0018] W. Eedus, J. Dean, and B. Zoph, “A review of sparse expert models in deep learning,” Sep. 2022, arXiv:2209.01667.

[0231]

[0019] J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, “Slimmable neural networks,” in Proc. IEEE Inti. Conf, on Learn. Repr. (ICLR), May 2019.

[0232]

[0020] J. Yu and T. S. Huang, “Universally slimmable networks and improved training techniques,” in Proc. IEEE / CVF Inti. Conf, on Computer Vision (ICCV), Oct. 2019, pp. 1803-181 1 .

[0233]

[0021] S. Chen, Y. Wu, Z. Chen, T. Yoshioka, S. Liu, J. Li, and X. Yu, “Don’t shoot butterfly with rifles: Multi-channel continuous speech separation with early exit transformer,” in Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), Jun. 2021 , pp. 6139-6143.

[0234]

[0022] D. Bralios, E. Tzinis, G. Wichem, P. Smaragdis, and J. L. Roux, “Latent iterative refinement for modular source separation,” in Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), Jun. 2023, pp. 1-5.

[0235]

[0023] X. Wang, Z. Chen, Y. Shi, J. Wu, N. Kanda, and T. Yoshioka, “Breaking trade-offs in speech separation with sparsely-gated mixture of experts,” May 2022, arXiv:221 1 .06493.

[0236]

[0024] M. Elminshawi, S. R. Chetupalli, and E. A. P. Habets, “Slim-Tasnet: A slimmable neural net- work for speech separation,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), Oct. 2023, pp. 1-5.

[0237]

[0025] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in Proc. IEEE Inti. Conf, on Learn. Repr. (ICLR), Apr. 2017.

[0238]

[0026] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR-halfbaked or well done?” in Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 626-630.

[0239]

[0027] M. Kolbsek, D. Yu, Z.-H. Tan, and J. Jensen, “Multitalker speech separation with utterance- level permutation invariant training of deep recurrent neural networks,” IEEE / ACM Trans. Audio, Speech, Lang. Process., vol. 25, no. 10, pp. 1901-1913, Oct. 2017.

[0240]

[0028] J. R. Hershey, Z. Chen, J. L. Roux, and S. Watanabe, “Deep clustering: Discriminative embed- dings for segmentation and separation,” in Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), Mar. 2016, pp. 31 -35.

[0241]

[0029] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization, ”in Proc. IEEE Inti. Conf, on [ .earn. Repr. (ICLR), May 2015, pp. 1 -15.

[0101] Yizeng Ilan, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang, “Dynamic neural networks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 1 1 , pp. 7436-7456, Nov. 2022.

[0242]

[0102] Noam Shazeer, * Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Ee, Geoffrey Hin- ton, and Jeff Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in Proc. IEEE Inti. Conf, on Learn. Repr. (ICLR), Apr. 2017.

[0243]

[0103] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024.

[0244]

[0104] Zhao You, Shulin Feng, Dan Su, and Dong Yu, “Speech-MoE: Scaling to large acoustic models with dynamic routing mixture of experts,” in Proc. Interspeech Conf., Aug. 2021 , pp. 2077-2081.

[0245]

[0105] Xiaofei Wang, Zhuo Chen, Yu Shi, Jian Wu, Naoyuki Kanda, and Takuya Yoshioka, “Break- ing trade-offs in speech separation with sparsely-gated mixture of experts,” May 2022, arXiv:221 1.06493.

[0246]

[0106] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama, “Adaptive neural net- works for efficient inference,” in Proc. Inti. Conf, on Machine Learning (ICML), Aug. 2017, vol. 70, pp. 527-536.

[0247]

[0107] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez, “Skipnet: Learning dynamic routing in convolutional networks,” in Proc. European Conf, on Computer Vision (ECCV), September 2018.

[0248]

[0108] Sanyuan Chen, Yu Wu, Zhuo Chen, Takuya Yoshioka, Shujie Liu, Jinyu Li, and Xiangzhan Yu, “Dont shoot butterfly with rifles: Multi-channel continuous speech separation with early exit transformer,” in Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), June 2021 , pp. 6139-6143.

[0249]

[0109] Dimitrios Bralios, Efthymios Tzinis, Gordon Wichern, Paris Smaragdis, and Jonathan Le Roux, “Latent iterative refinement for modular source separation,” in Proc. IEEE Inti. Conf, on Acoustics, Speech and Signal Processing (ICASSP), June 2023, pp. 1-5.

[0250]

[0110] Xiaohuai Le, Tong Lei, Kai Chen, and Jing Lu, “Inference skipping for more efficient real- time speech enhancement with parallel runs,” IEEE / ACM Trans. Audio, Speech, Lang. Process., vol.

[0251] 30, pp. 241 1-2421 , 2022.

[0111] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang, “Slimmable neural networks,” in Proc. IEEE Inti. Conf, on Learn. Repr. (ICLR), May 2019.

[0252]

[0112] Mohamed Elminshawi, Srikanth Raj Chetupalli, and Emanul A. P. Habets, “Slim-Tasnet: A slimmable neural network for speech separation,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), Oct. 2023, pp. 1-5.

[0113] Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang, “Dynamic slimmable network,” in Proc. ILEE / CVF Coni'. On Computer Vision and Pattern Recog- nition (CVPR), June 2021 , pp. 8607-8617.

Claims

Claims1 . An apparatus ( 100, 200, 300, 400, 500) comprising: an input interface ( 101) to receive an input segment (102) of an input sequential data; a neural network, NN, processor ( 1 10) to derive an output result (104) by pro- cessing the input segment (102) through a NN having a number of layers ( 1 1 1 ) from a first layer to a last layer, the NN using a predetermined number of deactivatable units which are selectively deactivatable; a gating module (120) configured to deactivate at least one deactivatable unit based on the input segment ( 102) and / or on at least one intermediate output segment (1 12, 123), an output interface (103) configured to provide an output (104) derived from the last layer of the NN.

2. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to decide how many deactivatable units are deactivated based on the input seg- ment (102) and / or on at least one intermediate output segment (1 12, 123).

3. The apparatus of claim 2, wherein the gating module (120) is configured, for at least one deactivatable unit, to decide how many deactivatable units to deactivate deter- ministically.

4. The apparatus of claim 2, wherein the gating module (120) is con figured, for at least one deactivatable unit, to decide how many deactivatable units to deactivate ran- domly.

5. The apparatus of any of the preceding claims, wherein the gating module ( 120) is configured to decide which deactivatable units to deactivate.

6. The apparatus of claim 5, wherein the gating module (120) is configured to decide which deactivatable units to deactivate deterministically.

7. The apparatus of claim 5 or 6, wherein the gating module ( 120) is configured to de- cide which deactivatable units to deactivate deterministically based on a ranking on the de- activatable units.

8. The apparatus of claim 7, wherein the gating module ( 120) is configured to decide which deactivatable units being neurons to deactivate based on the weight values of the connections entering in, and / or exiting out, each deactivatable unit, so as to provide a lower ranking to the deactivatable units that have comparatively low weight values of the connections entering and / or exiting than deactivatable units that have comparatively high weight values of the connections entering and / or exiting.

9. The apparatus of claim 7 or8, wherein the gating module (120) is configured to de- cide which deactivatable units to deactivate based on their importance quantified by a score function of at least the weight values of the connections entering in and / or exiting out the deactivatable units being neurons.

10. The apparatus of claim 7, wherein the gating module ( 120) is configured to decide which deactivatable units being neurons or patterns to deactivate based on the weighted values inputted to the deactivatable units or outputted by the deactivatable units, so as to provide a lower ranking to the deactivatable units that have comparatively low weighted inputted values or lower weighted outputted values than deactivatable units that have com- paratively high weight values entering have comparatively high weighted inputted values or high weighted outputted values.1 1 . The apparatus of claim 7, wherein the gating module ( 120) is configured to decide which deactivatable units to deactivate, the deactivatable units being connections between neurons or patterns and having weights, so as to deactivate the deactivatable units based on the weight values of the deactivatable units being connections, so as to provide a lower ranking to the deactivatable units being connections that have comparatively low weight values than deactivatable units being connections that have comparatively high weight val- ties.

12. The apparatus of claim 5, wherein the gating module (120) is configured, for at least one group of deactivatable units, to decide which deactivatable units to deactivate randomly.

13. The apparatus of any of the preceding claims, wherein the gating module (120) is configured, for at least one group of deactivatable units, to apply a utilization factor, wherein the deactivatable units of at least one group of deactivatable units are ranked from a first-ranked deactivatable unit to a last-ranked deactivatable unit, wherein a compara- tively high utilization factor implies a comparatively high number of activated deactivata- ble units including lower ranked deactivatable units, and a comparatively low utilization factor implies a comparatively low number of activated deactivatable units, wherein the deactivatable units are activated, in accordance to their decreasing ranking, at the increase of the utilization factor.

14. The apparatus of claim 13, wherein the ranking is predefined.

15. The apparatus of claim 13, wherein the ranking of the deactivatable units being neurons is at least partially conditioned by weight values of the connections entering in and / or exiting out the deactivatable units of at least one group of deactivatable units.

16. The apparatus of claim 13, wherein the ranking is derived by reordering the deacti- vatable units based on a score function of the weight values of the connections entering in and / or exiting out the deactivatable units, the dcactivatable units being neurons.

17. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to apply one single utilization factor to all, or at least to a plurality of, deactivat- able units of the NN.

18. The apparatus of any of the preceding claims, the deactivatable units being neurons or patterns and each deacti vatable unit being in a specific layer, wherein the gating module (120) is configured to apply, based on the input segment (102), a layer-specific utilization factor to each layer or to apply a layer-specific utilization factor to each group o f layers.

19. The apparatus of any claims 1 - 18, wherein at least one group of deactivatable units is formed by deactivatable units controlled with the same utilization factor, each group of deactivatable units being formed by deactivatable units in a same layer(s), but different groups have no deactivatable units in the same layer(s).

20. The apparatus of any of claims 1 - 16, the deactivatable units being connections be- tween neurons, and each deactivatable unit being associated with a group of deactivatable units entering in one particular layer or exiting from one particular layer, wherein the gat- ing module (120) is configured to apply, based on the input segment (102), a layer-specific utilization factor to each layer or to apply a group-specific utilization factor to each group of deactivatable units.21 . The apparatus of any of the preceding claims, wherein the gating module ( 120) is configured to apply, to at least one deactivatable unit or a group of deactivatable units, a unit-specific or group-specific utilization factor based at least on an intermediate output segment outputted by at least one of the preceding deactivatable units.

22. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to apply, to at least one deactivatable unit or a group of deactivatable units, a group-specific utilization factor based both on at least one intermediate output segment outputted by at least one of the preceding layers and on the input segment ( 102).

23. The apparatus of any of the preceding claims, wherein the input segment ( 102) is in a time-based sequence.

24. The apparatus of any of the preceding claims, wherein the input segment (102) is in a position-based sequence.

25. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to select between deactivating a second number of deactivatable units and deac- tivating a first number of deactivatable unit which is 0 or greater than 0, wherein the sec- ond number of deactivatable units is greater than the first number of deactivatable units.

26. The apparatus of any of the preceding claims, wherein the gating module ( 120) is configured to classify the input segment ( 102), based at least on the input segment (102), between at least a first classification result, indicative of the input segment ( 102) requiring a processing at a first computational complexity, and a second classification result, indica- tive of the input segment ( 102) requiring a processing at a second computational complex- ity which is higher than the first computational complexity, wherein the gating module ( 120) is configured to deactivate a greater number of deactivatable units in the case of the first classification result than in the case of the second classification result.

27. The apparatus of any of the preceding claims, wherein the gating module ( 120) is configured to classify the input segment ( 102), based at least on the input segment (102), between a plurality of classification results indicative of different computational complexi- ties, wherein the gating module (120) is configured to deactivate a greater number of deac- tivatable units for input segment(s) having classification result(s) indicative of the input segment (102) requiring a processing at comparatively low computational complexity than for input segment(s) (102) having classification result(s) indicative of the input segment (102) requiring a processing at comparatively high computational complexity.

28. The apparatus of any of the preceding claims, wherein the gating module ( 120) is configured to classify the input segment (102), based at least on the output, between at least a first classification result, indicative of the input segment (102) requiring a pro- cessing at a first computational complexity, and a second classification result, indicative of the input segment ( 102) requiring a processing at a second computational complexity which is higher than the first computational complexity, wherein the gating module ( 120) is configured to deactivate a greater number of deactivatable units for at least one group of deactivatable units in the case of the first classification result than in the case of the sec- ond classification result.

29. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to classify the input segment (102), based at least on the output result, between a plurality of classification results indicative of different computational complexities, wherein the gating module ( 120) is configured to deactivate a greater number of deactivat- able units for input segment(s) having classification result(s) indicative of comparativelylow computational complexity than for input segment(s) having classification result(s) in- dicative of comparatively high computational complexity.

30. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to classify the input segment ( 102), based at least on an intermediate output re- sult, between at least a first classification result, indicative of the input segment ( 102) re- quiring a processing at a first computational complexity, and a second classification result, indicative of the input segment (102) requiring a processing at a second computational complexity which is higher than the first computational complexity, wherein the gating module (120) is configured to deactivate a greater number of deactivatable units for at least one group of deactivatable units in the case of the first classification result than in the case of the second classification result.

31. The apparatus of any of the preceding claims, wherein the gating module ( 120) is configured to classify the input segment ( 102), based at least on an intermediate output re- sult, between a plurality of classification results indicative of different computational com- plexities, wherein the gating module ( 120) is configured to deactivate a greater number of deactivatable units for an input segment (102) having a classification result indicative of comparatively low computational complexity than for an input segment (102) having a classification result indicative of comparatively high computational complexity.

32. The apparatus of any of claims 26-31 , wherein the gating module ( 120) is config- ured to classify the input segment ( 102) based at least on a noisiness measurement of the input segment (102), or a processed version thereof, between at least the first classification result, indicative of comparatively low noisiness, and the second classification result, in- dicative of comparatively high noisiness, so that the gating module (120) deactivates a greater number of deactivatable units for at least one group of deactivatable units in the case of the comparatively low noisiness than in the case of the comparatively high noisi- ness.

33. The apparatus of any of claims 26-31 , wherein the gating module (120) is config- ured to classi fy the input segment (102) based at least on a noisiness measurement of the input segment ( 102), or a processed version thereof, between a plurality of classificationresults indicative of different noisinesses, wherein the gating module ( 120) is configured to deactivate a greater number of deactivatable units for input segment(s) having classifica- tion result(s) indicative of low noisiness than for input segment(s) having classification re- sults) indicative of comparatively high noisiness.

34. The apparatus of any of claims 26-33, wherein the gating module ( 120) is config- ured to classify the input segment (102) based on a harmonicity measurement of the input segment (102), or a processed version thereof, between at least the first classification re- sult, indicative of comparatively high harmonicity, and the second classification result, in- dicative of comparatively low harmonicity, so that the gating module ( 120) deactivates a greater number of deactivatable units for at least one group of deactivatable units in the case o f comparatively high harmonicity than in the case of the comparatively low harmon- icity.

35. The apparatus of any of claims 26-34, wherein the gating module (120) is config- ured to classify the input segment (102) based at least on a harmonicity measurement of the input segment (102), or a processed version thereof, between a plurality of classifica- tion results indicative of different harmonicities, wherein the gating module ( 120) is con- figured to deactivate greater number(s) of deactivatable units for input segment(s) having classification result(s) indicative of comparatively high harmonicity than for input segments) having classification result(s) indicative of comparatively low harmonicity.

36. The apparatus of any of claims 26-35, wherein the gating module ( 120) is config- ured to classify the input segment (102) based on an interference measurement between at least the first classification result, indicative of comparatively low interference, and the second classification result, indicative of comparatively high interference, so that the gat- ing module ( 120) deactivates a greater number of deactivatable units for at least one group of deactivatable units in the case of the comparatively low interference than in the case of the comparatively high interference.

37. The apparatus of any of claims 26-36, wherein the gating module ( 120) is config- ured to classify the input segment ( 102) based at least on an interference measurement ofthe input segment (102), or a processed version thereof, between a plurality of classifica- tion results indicative of different levels of interference, wherein the gating module (120) is configured to deactivate a greater number of deactivatablc units for an input segment (102) having a classification result indicative of comparatively low interference than for an input segment (102) having a classification result indicative of comparatively high interference.

38. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to classify the input segment (102), based at least on an additional signal, be- tween at least a first classification result, indicative of the input segment ( 102) requiring a processing at a first computational complexity, and a second classification result, indica- tive of the input segment (102) requiring a processing at a second computational complex- ity which is higher than the first computational complexity, wherein the gating module (120) is configured to deactivate a greater number of deactivatable units for at least one group of deactivatable units in the case of the first classification result than in the case of the second classification result.

39. The apparatus of any of the preceding claims, wherein the gating module (120) is configured to classify the input segment (102), based at least on an additional signal, be- tween a plurality of classification results, indicative of the input segment (102) requiring processings at different computational complexities, wherein the gating module (120) is configured so that classification results indicative of the input segment (102) requiring pro- cessings at comparatively reduced computational complexities imply a deactivation of a comparatively increased number of units, and classification results indicative of the input segment (102) requiring processings at comparatively increased computational complexi- ties imply a deactivation of a comparatively reduced number of units, or no deactivation at all.

40. The apparatus of claim 38 or 39, wherein the input segment (102) is received from a bitstream, and the additional signal is obtained from signalling in the bitstream.

41. The apparatus of any of claims 26-40, configured to skip at least one by deactivat- ing all the deactivatable units of the layer, and bypassing the at least one layer.

42. The apparatus of any of claims 26-41 , configured to classify the inputs segment be- tween more than two classification results indicative of different levels of computational complexity required, the gating module ( 120) being configured to deactivate, for at least one group of deactivatable units, the deactivatable units in accordance with the classified level of computational complexity required, so that a higher computational complexity re- quired implies a lower number of deactivation of deactivatable units for the at least one group of deactivatable units, and a lower computational complexity required implies a higher number of deactivation of deactivatable units for the at least one group of deactivat- able units.

43. The apparatus of any of the preceding claims, wherein the NN is inputted with the additional signal in at least one of the layers.

44. The apparatus of any of the preceding claims, wherein the gating module (120) in- cludes a second NN, or another learning unit, to classify the input segment (102).

45. The apparatus of any of the preceding claims, wherein the gating module (120) in- cludes a deterministic classifier unit to classify the input segment (102).

46. The apparatus of any of the preceding claims, wherein the NN is trained to reduce noise from the input segment (102) being indicative of an audio signal, so as to provide the output result as a denoised version of the input segment (102).

47. The apparatus of any of the preceding claims, wherein the NN is trained to extract a target component from the input segment (102) being indicative of an audio signal, so as to provide the output result as a target component of the input segment (102).

48. The apparatus of any of the preceding claims, wherein the NN is trained to extract a speech component from the input segment (102) being indicative of an audio signal, so as to provide the output result as a speech component of the input segment (102).

49. The apparatus of any of the preceding claims, wherein the NN is trained to extract a component from one specific audio source from the input segment (102) being indicativeof an audio signal with multiple audio sources, so as to provide the output result as a com- ponent of the audio signal from the specific audio source.

50. The apparatus of any of the preceding claims, wherein the NN is trained to provide the output result as having a higher dimensionality than the dimensionality of the input segment ( 102).51 . The apparatus of any of the of the preceding claims, wherein the deactivatable units include neurons.

52. The apparatus of any of the of the preceding claims, wherein the deactivatable units include patterns, the NN being a convolutional NN.

53. The apparatus of any of the of the preceding claims, wherein the deactivatable units include bias.

54. The apparatus of any of the of the preceding claims, wherein the deactivatable units include connections carrying weights.

55. The apparatus of any of the of the preceding claims, wherein the deactivatable units include input elements of the input segment (102).

56. The apparatus of any of the preceding claims, wherein the deactivatable units include learnable parameters of the NN.

57. The system of any of the preceding claims configured for performing audio filtering the input segment ( 102) representing an audio signal , wherein the NN is configured to pro- duce at least one output segment by processing the input signal.

58. The system of any of the preceding claims configured to choose among operating at a first mode and operating at a second mode, wherein the first mode implies controlling the deactivation of the deactivatable units according to a first criterion, and the second modeimplies controlling the deactivation of the deactivatable units according to a second crite- rion59. The system of clam 58, wherein the choice between the first mode and the second mode is performed deterministically60. A method comprising: receiving an input segment of an input sequential data through a neural network, NN, deriving an output result by processing the input segment through the NN having a number of layers from a first layer to a last layer, the NN having a prede- termined number of deactivatable units; deactivating at least one deactivatable unit based on the input segment and / or on at least one intermediate output segment; outputting an output result derived from the last layer of the NN.61 . A non-transitory storage unit storing instructions which, when executed by a pro- cessor, cause the processor to perform the method of claim 60 or at least one of the steps of the method of claims 60.

Images