Neural network processor and neural network processing method

Abstract

A neural network processor is designed to process sequential windows (W1, W2,... Wn) of a time-dependent signal. Each window contains multiple samples (ns) of the signal over a time interval (T), with each window shifted relative to the previous one by a time-step (ΔT) smaller than T. This time-step corresponds to a base shift amount (h) defined by h=[ΔT / T ns]. The processor executes a neural network with multiple layers (L), each containing a plurality of neurons (N(...,Y,...L)), indexed by time domain (Y). It performs operations for each sample window, including computing a differential result signal for a neuron by referencing a neuron with an index value (Y) determined by the base shift amount and an accumulated up / down-sampling factor S.

Description

[0001] NEURAL NETWORK PROCESSOR AND NEURAL NETWORK PROCESSING

[0002] METHOD

[0003] CLAIM OF PRIORITY

[0004] This application claims the benefit of priority to European Patent Application Serial No. 23306052.4, filed on June 28, 2023, which is incorporated herein by reference in its entirety.

[0005] TECHNICAL FIELD

[0006] The present disclosure pertains to neural network processors. The present disclosure further pertains to a neural network processing method.

[0007] BACKGROUND

[0008] Neural networks become more and more advanced, and one of their applications is in the field of processing time-dependent signals, e.g. signals representing a video or a sound. In operation the neural network processes subsequent windows of the time dependent signal. Each window comprises a plurality of samples of the time dependent signal for subsequent points in time in a time-interval. Each window of samples is shifted in time relative to its immediately preceding window with a time-step that is smaller than the timeinterval over which the window extends.

[0009] In processing time dependent signals it is necessary that the throughput of the neural network processor is sufficient in order to match the rate at which the time dependent signal is received. It is however also desired that this can be achieved without undue computational effort.

[0010] SUMMARY According to a first aspect of the present disclosure a neural network processor is provided for efficiently processing subsequent windows of a time dependent signal.

[0011] According to a second aspect of the present disclosure a neural network processing method is provided for efficiently processing subsequent windows of a time dependent signal.

[0012] The present disclosure further pertains to a tangible or non-transitory computer-readable medium having computer-executable instructions stored thereon that, when executed by a processor, perform the method.

[0013] Embodiments of the neural network processor of the present disclosure are configured to execute a neural network with a plurality of neural network layers.

[0014] Each layer has a respective plurality of neurons of which the output is computed as a function of a set of one or more inputs. Typically, the set of inputs corresponds to a kernel in a previous layer or in the window being processed, wherein the kernel is centered around a position with coordinates corresponding to the coordinates of the neuron of which the output is being computed.

[0015] E.g. if the window comprises a vector of audio signal values a(0),a(l),...,a(n-l) the input layer of the neural network may perform a convolution with a convolution kernel such that the output values o(j) of the neurons n(j) are determined as:

[0016] Therein 2w+l is the size of the kernel

[0017] It may be avoided that the value j+i is outside the boundary of the window by performing the addition modulo the size of the window or by skipping the contributions of non-existing input elements.

[0018] Due to the fact that the window comprises a series of data elements, the audio signal values, ordered in time, the neurons o(j) as specified above also have a time-related ordering. I.e. the index according to which they are addressable is a time domain related index. This also applies if the layer applies a scaling. For example if a down-scaling is applied with a factor s, e.g. s=2, then the output values o(j) of the neurons n(j) are determined as: This further applies if the function with which the output values are computed is not a linear function as in the previous examples.

[0019] More generally, in the one-dimensional case, the output value of a neuron o(j) may be written as o(j) = f a(sj — w), a(sj — w + 1), ..., a sj + w)).

[0020] Typically the neural network has a set of successive layers and size of the layers determined for the time-domain related index reduces. For example, the set of successive layers comprises convolutional layers that perform a convolution in the time-domain related index (TDRI) or pooling layers that perform a pooling. Due to the fact that the output of a layer is a series of data elements having an ordering along a time domain related index, also the neurons of the subsequent layer have an ordering along a time-domain related index.

[0021] The same principle is applicable to processing of higher dimensional time based signals.

[0022] For example, in case the input signal is a multi-dimensional signal b(t,x,y,z), the output o(t’,x’,y’,z’) of a neuron n(t’,x’,y’,z’) of the input layer may be computed with a convolution like:

[0023] Therein 2wt+l, 2wx+l, 2wy+l and 2wz+l are the sizes of the kernel along the time-axis t and the axis x,y,z for the further dimensions of the signal. Likewise, in this case, the neurons of the input layer are addressable with their time-domain related index t’, and additionally with their three other index axis x’,y’,z’. This is equally applicable if the layer performs a scale-down operation or a scale-up operation as specified in the example below. Also this is applicable if the function with which the output of the neuron is computed is not a linear operation, but an arbitrary function of the input values within the kernel.

[0024] A neural network layer has a respective size corresponding to the address range for each index. As specified above, the neural network function computes an output value of a neuron based on the values in the input signal that are within the address range of the kernel for that neuron. Subsequent neurons within an address range of a neuron index, for example the time-domain based index, correspond to kernels in the input signal domain having a center at subsequent positions in the coordinate system of the input signal, for example having subsequent time values in this example. If the window of input signals spans a time interval T then the address range of the time-domain based index also spans that time interval T. This implies that each step along the timedomain based index corresponds to a step in time equal to the value T divided by the address range of the time-domain based index. In case the layer performs a scale-down operation the timestep between subsequent index value of the timedomain related index increase. In case the layer performs a scale-up operation, as in a deconvolution layer, the timestep between subsequent index value of the time-domain related index decrease.

[0025] In case the neural network layer is a further layer, i.e. an intermediary layer or an output layer, then the same principle applies, unless the further layer receives a cropped portion of the preceding layer. In the latter case, a time interval can be computed that correspond to the cropped portion, and the timestep is equal to span of time of the cropped portion divided by the address range of the time-domain based index for that layer.

[0026] In some embodiments the neural network implemented by the neural network processor comprises a convolutional section and a deconvolutional section, for example configured in a U-net. In the convolutional section a dimension reduction takes place, wherein the neural network extracts features from the input and in the deconvolutional section the dimension is gradually increased to the dimensions of the input signal. In the deconvolutional section the neural network determines how the extracted features are identified in the input signal. The U-net may have bridge-sections between the convolutional section and the deconvolutional section.

[0027] Alternatively, the neurons of a layer may be addressable with one or more additional indices. For example, the input signal is obtained from a Short-Time Fourier Transform (STFT), the input layer is addressable with an additional index, which may be denoted as the frequency domain related index. As a further example the input is a 2D-video signal. In this case the neurons of a layer are addressable with two additional indices, which may be denoted as spatial domain-related indices. In a still further example, the input is a 3D-video signal. In this case the neurons of a layer are addressable with three additional indices, also to be denoted as spatial domain-related indices. Arbitrary higher dimensional addressable configurations are conceivable, for example in case the input signal to be processed is a combination of an audio signal and a video signal.

[0028] As noted it is a challenge of neural network processing in the time-related domain to meet the rate at which input data is received. In order to address this challenge, the neural network processor of the present disclosure is configured to perform respective operation cycles for respective windows of samples. An operation cycle comprises: computing respective result signal values of neurons of a layer; computing differential result signal values of the neurons from the result signal values obtained in the current operation cycle and result signal values obtained in a preceding operation cycle; providing respective output signal values for respective ones of the neurons of the layer of which the differential result signal values exceed a threshold value; computing respective result signal values of neurons of a subsequent layer based on the provided output signal values.

[0029] In an above-mentioned operation cycle the differential result signal value of a particular neuron having a particular time domain related index is computed as the difference between the result signal value of the particular neuron obtained in the current operation cycle and the result signal value of a reference neuron obtained in the preceding operation cycle and which reference time domain related index Y’ is related to the particular time-domain related index as:

[0030] Y' = (Y + [h ■ SJ) mod sY where mod is the modulo operation, h is the base shift amount of the window, being h = and S is the accumulated up / down- sampling factor at the output of the layer L being the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y preceding window of samples if the reference time domain related index ( Y’ ) is inside the time domain related index range of the layer L. The neural network processor performs a respective operation cycle for each window of the input signal. The operations may be performed in sequential manner. For example, for each received window the neural network processor subsequently performs the steps for the data received in that window. This implies that the processor has available respective timeframes within the duration of the window for performing the operations in each of the layers.

[0031] Alternatively, the operations may be performed in a pipe-lined manner, wherein the neural network processor simultaneously performs operations in a lower ranked layer for more recently received input data and in a higher ranked layer for the older data.

[0032] BRIEF DESCRIPTION OF THE DRAWINGS

[0033] These and other aspects of the present disclosure are elucidated in more detail with reference to the drawings. Therein

[0034] FIG. 1 schematically shows an embodiment of a neural network processor;

[0035] FIG. 2 schematically shows an embodiment of a system comprising an embodiment of the neural network processor;

[0036] FIG. 3 shows an intermediate processing result in the system of FIG. 2; and

[0037] FIG. 4 shows an example of the neural network processor for use in the system of FIG. 2.

[0038] DETAILED DESCRIPTION

[0039] FIG. 1 schematically shows a neural network processor 1 for processing subsequent windows Wl, W2, ...Wn of a time dependent signal. Each window comprises a plurality ns of samples of the time dependent signal for subsequent points in time in a time-interval. By way of example a window comprises 64 acoustic signal values sampled at subsequent points in time, for example each 1 ms. Therewith a window with 64 samples spans a time-interval of 64 ms. Each succeeding window to be processed by the neural network processor 1 is shifted in time relative to the window being succeeded therewith. The time-step AT is smaller than the time-interval T and preferably corresponds to a number of samples, denoted as the base shift amount h, being an integer power of 2, wherein the integer power is for example in the range of 1 to 5. For example if a time step is taken of 4ms, the base shift amount is 4, corresponding to the power 2 of 2.

[0040] The neural network processor 1 is configured to execute a neural network with a plurality of neural network layers L. In the example shown, the neural network has 5 convolutional layers L = 1,...,5. It is noted that the neural network processor 1 may have a physical architecture that closely corresponds to the logical architecture of the neural network but that is not necessary. It would even be possible to implement the neural network with a single general purpose processor that sequentially performs all operations of the neural network. In typical cases, the neural network is implemented by a multiprocessor system, e.g. a GPU, or by a dedicated multicore system as specified in Patent Cooperation Treaty Patent Application Publication No. W02020 / 025680 and in United States Patent Application Pubheation No. US2021 / 0319295, each of which is incorporated herein by reference in its entirety.

[0041] The layers L=l,...,5 each have a respective plurality of neurons that are addressable with at least a time domain related index Y. In this example, the layers are further addressable with a channel index C. By way of example the address range for each index for each of the layers is shown in the following table.

[0042] In operation the neural network processor 1 cyclically performs a set of operations as specified below. The set of operations are repeated for each window of samples and comprise: a) computing respective result signal values R(Y,C,L) of neurons N(Y,C,L) of a layer L, b) determining differential result signal values D(Y,C,L) of the neurons N(Y,C,L) of the layer L, c) providing respective output signal values O(Y,C,L) for respective ones of the neurons N(Y,C,L) of the layer L of which the differential result signal values D(Y,C,L) exceed a threshold value;

[0043] A subsequent layer (if any) can then compute respective result signal values R(Y,C,L+1) of neurons N(Y,C,L+1) based on the provided output signal values. If there is no subsequent layer the output signal values serve as the result of the neural network.

[0044] The first mentioned step a) may comprise conventional neural network computation operations, such as convolution operations.

[0045] The second operation b) differs from a conventional approach as described below. As specified in further detail therein, contrary to a conventional approach, the differential result signal value D(Y,C,L) of a particular neuron N(Y,C,L) having a time-domain related index Y is computed dependent on a value of reference neuron in the layer L that has a reference time domain related index Y’ that is not necessarily equal to the value Y. More specifically, the differential result signal value D(Y,C,L) is equal to the difference between the result signal value R(Y,C,L) of the particular neuron N(Y,C,L) obtained in the respective operation n and the result signal value R(Y’,C,L) of a reference neuron N(Y’,C,L) obtained in the respective operation for the preceding window of samples. The reference time domain related index Y’ of the reference neuron is computed from the time domain related index Y of the particular neuron by adding a shift amount to the time domain related index Y. Therein the shift amount is equal to the base shift amount h of the window (i.e. the number of samples corresponding to the time shift AT and the accumulated up / down-sampling factor S at the output of the layer L, which is the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y. The base shift amount h of the window is computed with If the computed shift amount is not integer, it may be rounded to the nearest integer value to compute the reference time domain related index Y’. In case the computed reference time domain related index Y’ would fall outside the address range of the layer the computation of the differential result signal value D(...,Y,...,L) of the particular neuron N(...,Y,...,L) may be skipped or based on an estimated reference value.

[0046] In a practical embodiment the reference time domain related index Y’ is computed as:

[0047] Y' = (Y + [h ■ SJ) mod sY where mod is the modulo operation Therewith the computed reference time domain related index Y’ is always inside the time domain related index range of the layer.

[0048] In step c) output signal values O(Y,C,L) are selectively provided for respective ones of the neurons N(Y,C,L) of the layer L of which the differential result signal values D(Y,C,L) exceed a threshold value. Therewith the number of output signal values is reduced (Sparsified). As the number of output signal values of a layer is typically less than the number of result signal values of the layer the amount of data to be communicated from the layer L to the layer L+l is reduced. Also the computational load involved for computing the output values of the subsequent layer L+l is reduced. Due to the fact that the differential result signal value D(Y,C,L) of a particular neuron N(Y,C,L) having a time-domain related index Y is computed dependent on a value of reference neuron having a time-domain related index that is computed in accordance with the above- mentioned principles, result signal values R(Y,C,L) and R(Y’,C,L) are strongly correlated so that indeed the sparsification is effective.

[0049] In the example shown, the windows WO, Wl,...,Wn,... each comprise a number of ns = 64 samples spanning a time interval T of 64 ms and the base shift amount h = 4.

[0050] The first convolutional layer, L=l, has a range of 16 for the time-domain related index and a range of 4 for the channel index. The result signal values R(Y,C,1) of the neurons N(Y,C,1) of the first layer may be computed in a conventional manner, e.g. by a convolution with a respective convolution kernel for each channel. In order to compute the differential signal values, the time domain related index shift hl to be applied is computed as the product of the base shift amount h (=4) and the accumulated up / down-sampling factor S at the output of the layer L, which is the ratio sY / ns = 1 / 4. Hence, the product hl is 1. Accordingly, the differential signal value Dt(Y,C,l) of a neuron N(Y,C,1) at time t is computed as Rt(Y,C,l) - Rt-i(Y’,C,l), wherein Y’ = Y+l, provided that the value Y’ is within the time domain related index range of the layer.

[0051] For the remaining layers 2 to 5, the time domain related index shift is truncated to 0, so that in practice the differential signal value Dt(Y,C,L) of neurons N(Y,C,L) of these layers at time t are computed in a conventional manner as Rt(Y,C,L) - Rt-i(Y,C,L).

[0052] As noted in the introductory part, it is not necessary that the timedependent signal to be processed by the neural network processor 1 is a scalar. By way of example FIG. 2 shows a system comprising a Short-Time Fourier Transform (STFT) unit 2 and a neural network processor 1 as disclosed herein. The STFT unit 2 receives at its input an audio signal M(t), which is a timedependent scalar and provides at its output a power spectrogram S(x,y) of which an example is illustrated in FIG. 3, each sample is a vector Sy(x) of values, wherein each value represents the magnitude of the original audio signal power in a frequency band. Presuming that the audio spectrogram represents the spectrum within a frequency range [0, fmax), then a value x corresponds to a frequency band

[0053] Therein ms is the number of elements in the vector Sy(x).

[0054] The windowing unit 3 repeatedly selects a window of samples from the STFT unit 2 and provides this to the neural network processor 1. By way of example each window comprises 128 samples in a time-interval T of approximately 11 seconds. Each sample is a spectrogram with 512 spectral signal values. Each window of samples is shifted in time relative to its immediately preceding window with a time-step AT (also denoted as hop length), which timestep is smaller than the time-interval T. The timestep corresponds to a number of samples, which is denoted herein as base shift amount h. For practical purposes the base shift amount h is an integer power of 2. The integer power is for example in the range of 1 to 5, but is not limited thereto. In an example, the neural network processor 1 comprises a series of convolutional layers, similar to the arrangement as shown in FIG. 1 to extract a stream F(t) of feature data.

[0055] The convolutional layers for this application differ from those of the embodiment as shown in FIG. 1 that the neurons therein are additionally addressable with a frequency domain related index. For this purpose the neurons of the neural network implemented by the neural network processor 1 will be denoted as N(X,Y,C,L). The improvement in this example is achieved by analogously taking into account a time-domain related shift to compute the reference time domain related index.

[0056] A different example is now elucidated in more detail with reference to FIG. 4. In this example, the neural network has a U-Net architecture, as disclosed in Proceedings of the 18th ISMIR Conference, Suzhou, China, October 23-27, 2017, which in this case has layers L = 1, 2,..., 11. The layers L =1,...,6 are convolutional layers and the subsequent layers 7,.., 11 are deconvolution al layers. It is noted that the neural network processor 1 may have a physical architecture that closely corresponds to the logical architecture of the neural network but that is not necessary. It would even be possible to implement the neural network with a single general purpose processor that sequentially performs all operations of the neural network. In typical cases, the neural network is implemented by a multiprocessor system, e.g. a GPU, or by a dedicated multicore system as specified in Patent Cooperation Treaty Patent Application Publication No. W02020 / 025680 and in United States Patent Application Publication No. US2021 / 0319295.

[0057] The layers L=l,...,ll each have a respective plurality of neurons that are addressable with at least a time domain related index Y. In the embodiment shown, the layers are further addressable with a frequency domain related index X and a channel index C. The address range for each index for each of the layers is shown in the following table.

[0058]

[0059] In operation the neural network processor 1 cyclically performs a set of operations as specified below. The set of operations are repeated for each window of samples and comprise: a computing respective result signal values R(X,Y,C,L) of neurons N(X,Y,C,L) of a layer L, b determining differential result signal values D(X,Y,C,L) of the neurons N(X,Y,C,L) of the layer L, c providing respective output signal values O(X,Y,C,L) for respective ones of the neurons N(X,Y,C,L) of the layer L of which the differential result signal values D(X,Y,C,L) exceed a threshold value;

[0060] In turn the subsequent layer (if any) will then compute respective result signal values R(X,Y,C,L+1) of its neurons N(X,Y,C,L+1) based on the provided output signal values. Otherwise the output signal values may serve as the output of the neural network as such.

[0061] As mentioned before, the first mentioned step a) may comprise conventional neural network computation operations, such as convolution operations (for layers 1 to 6) and deconvolution operations (for layers 7 to 11). Also as shown by block arrows in FIG. 1, the deconvolution layers may concatenate their outputs with those of a respective convolutional layer having the same output ranges. Similarly as specified in the first mentioned example, the second operation b) differs from a conventional approach in that the differential result signal value D(X,Y,C,L) of a particular neuron N(X,Y,C,L) having a time-domain related index Y is computed dependent on a value of reference neuron in the layer L that has a reference time domain related index Y’ that is not necessarily equal to the value Y. More specifically, the differential result signal value D(X,Y,C,L) is equal to the difference between the result signal value R(X,Y,C,L) of the particular neuron N(X,Y,C,L) obtained in the respective operation n and the result signal value R(X,Y’,C,L) of a reference neuron N(X,Y’,C,L) obtained in the respective operation for the preceding window of samples. The reference time domain related index Y’ of the reference neuron is computed from the time domain related index Y of the particular neuron by adding a shift amount to the time domain related index Y. Therein the shift amount is equal to the base shift amount h of the window (i.e. the number of samples corresponding to the time shift AT and the accumulated up / down-sampling factor S at the output of the layer L, which is the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y. The base shift amount h of the window is computed with

[0062] If the computed shift amount is not integer, it may be rounded to the nearest integer value to compute the reference time domain related index Y’. In case the computed reference time domain related index Y’ would fall outside the address range of the layer the computation of the differential result signal value D(X,Y,C,L) of the particular neuron N(X,Y,C,L) may be skipped or based on an estimated reference value. In case the neural network has one or more deconvolution layers, then the reference time domain related index Y’ of the reference neuron is computed from the time domain related index Y of the particular neuron by adding a shift amount to the time domain related index Y, which shift amount is the same of that of the convolutional layer with which the output is concatenated.

[0063] In a practical embodiment the reference time domain related index Y’ is computed as:

[0064] Y = (Y + [h ■ 5J) mod sY where mod is the modulo operation Therewith the computed reference time domain related index Y’ is always inside the time domain related index range of the layer.

[0065] In step c) output signal values O(X,Y,C,L) are selectively provided for respective ones of the neurons N(X,Y,C,L) of the layer L of which the differential result signal values D(X,Y,C,L) exceed a threshold value. Therewith the number of output signal values is reduced (Sparsified). As the number of output signal values of a layer is typically less than the number of result signal values of the layer the amount of data to be communicated from the layer L to the layer L+l is reduced. Also the computational load involved for computing the output values of the subsequent layer L+l is reduced. Due to the fact that the differential result signal value D(X,Y,C,L) of a particular neuron N(X,Y,C,L) having a time-domain related index Y is computed dependent on a value of reference neuron having a time-domain related index that is computed in accordance with the above- mentioned principles, result signal values R(X,Y,C,L) and R(X,Y’,C,L) are strongly correlated so that indeed the sparsification is effective.

[0066] In the example shown, the windows WO, Wl,...,Wn,... each comprises a number of ns = 128 samples spanning a time interval T of approximately Il s. Suppose further that the base-shift amount is 8, which is an integer power of 2. The first convolutional layer, L=l, has a range of 64 for the time-domain related index, a range of 256 for the frequency-domain related index and a range of 16 for the channel index. The result signal values R(X,Y,C,1) of the neurons N(X,Y,C,1) of the first layer may be computed in a conventional manner, e.g. by a convolution with a respective convolution kernel for each channel. In order to compute the differential signal values, the time domain related index shift hl to be applied is computed as the product of the base shift amount h (=8) and the accumulated up / down-sampling factor S at the output of the layer L, which is the ratio sY / ns = 1 / 2. Hence, the product hl is 4.

[0067] Accordingly, the differential signal value Dt(X,Y,C,l) of a neuron N(X,Y,C,1) at time t is computed as Rt(X,Y,C,l) - Rt-i(X,Y’,C,l), wherein Y’ = Y+4, provided that the value Y’ is within the time domain related index range of the layer. For practical purposes, the reference time-domain related index is computed with the expression Y = (Y + [h ■ SJ) mod sY, so that always a time- domain related index value is obtained that is within the addressable range. The exemplary calculations apply equally to the deconvolution al layer 11, which has the same accumulated up / down-sampling factor at its output. It is noted that the computation of the absolute signal values R(X,Y,C,11) of the deconvolutional layer 11 can take place in a conventional manner.

[0068] The second convolutional layer, L=2, has a range of 32 for the time-domain related index, a range of 128 for the frequency-domain related index and a range of 32 for the channel index. The result signal values R(X,Y,C,2) of the neurons N(X,Y,C,2) of the second layer may likewise be computed in a conventional manner, e.g. by a convolution with a respective convolution kernel for each channel. In order to compute the differential signal values, the time domain related index shift h2 to be applied is computed as the product of the base shift amount h (=8) and the accumulated up / down-sampling factor S at the output of the layer L. In this case the value of S is 1 / 4. Hence, the product h2 is 2.

[0069] Accordingly, the differential signal value Dt(X,Y,C,2) of a neuron N(X,Y,C,2) at time t is computed as Rt(X,Y,C,2) - Rt-i(X,Y’,C,2), wherein Y’ = Y+2 (mod 32). The exemplary calculations apply equally to the deconvolutional layer 10, which has the same accumulated up / down-sampling factor at its output.

[0070] The third convolutional layer, L=3, has a range of 16 for the time-domain related index, a range of 64 for the frequency-domain related index and a range of 64 for the channel index. The result signal values R(X,Y,C,3) of the neurons N(X,Y,C,3) of the third layer may likewise be computed in a conventional manner, e.g. by a convolution with a respective convolution kernel for each channel. In order to compute the differential signal values, the time domain related index shift h3 to be applied is computed as the product of the base shift amount h (=8) and the accumulated up / down-sampling factor S at the output of the layer L. In this case the value of S is 1 / 8. Hence, the product h3 is 1.

[0071] Accordingly, the differential signal value Dt(X,Y,C,3) of a neuron N(X,Y,C,3) at time t is computed as Rt(X,Y,C,3) - Rt-i(X,Y’,C,3), wherein Y’ = Y+l (mod 16). The exemplary calculations apply equally to the deconvolutional layer 9, which has the same accumulated up / down-sampling factor at its output. For the layers 4 to 8, the time domain related index shift is truncated to 0, so that in practice the differential signal value Dt(X,Y,C,L) of neurons N(X,Y,C,L) of these layers at time t are computed as Rt(X,Y,C,L) - Rt-i(X,Y,C,L)

[0072] It is noted that the neural network processor may perform the computations as specified above for a specific layer L also for other neural network layers, but this is not necessary. An improvement in computational efficiency is already achieved if the measure is applied to a single neural network layer. In some examples a layer L’ is stateless. In that case, the neural network processor computes the output values of that layer L’ independent of previous input received by the layer.

[0073] For the neural network as shown in FIG. 4, it was investigated which selection of the base shift provides an optimal event sparsity. Results of this investigation are shown in the following table for the reference case, wherein no differential compensation is applied and for four selections of the base shift h. Therein the operational mode is shown in the first column. The number of events per operational cycle are shown in the second column and the sparsity as compared the reference case is shown in the third column. increase first with increasing base shift and with a further increase in base shift reduces again. In this example the highest sparsity was obtained with a base shift of 8. Also a high sparsity was obtained for a base shift of 4. Nevertheless, a base shift of 8 would be preferred in this case, also because this enable a throughput that is twice as high as that in case with the base shift of 4.

[0074] The pattern may be explained as follows. Clearly, more redundancy is present in mutually subsequent windows WO, Wl,...Wn,.. if the time step, and the corresponding base shift is smaller. On the other hand, a compensating shift (Y, Y’) of the time-related index can be applied in a larger number of layers if the corresponding base shift is larger. Presuming that the size of the layers in the direction of the time related index is reduced by 2 each subsequent layer then the number of layers wherein a proper compensation of the time step by a time related index shift is proportional to the 21og of the base shift. For layers wherein the reduction in size is reduced by an amount exceeding the base shift, the time related index shift would become a non-integer number between 0 and 1. As a result a neuron having a time-related index Y does not have a corresponding reference neuron with an optimally matching time-related index Y’. It is expect that more generally a high sparsity is achieved if the base shift is approximately equal to the square root of the size of the first layer in the direction of the time domain related index.

[0075] As noted above the base shift is preferably an integer power of 2. However, it may also be considered to select a power of another number, e.g. 3 if the size of the layers stepwise reduce by that other number.

[0076] As noted above, the present disclosure is also applicable to higher dimensional input signals. As an example the input is a sequence of windows, wherein each window comprises a sequence of image frames, and the neural network processes a complete window at a time.

[0077] In this case a neural network layer typically has in addition to the timedomain related index Y a first spatial coordinate related index IS1 and a second spatial coordinate related index IS2. Generally a layer also has a channel index. Hence, the neurons of a layer L are denoted as N(IS1,IS2,Y,C,L)

[0078] In this case the differential result signal value D(IS1,IS2,Y,C,L) of a neuron N(IS1,IS2,Y,C,L) is computed as the difference between the output value O(IS1,IS2,Y,C,L) at point in time t and the reference output value O(IS1,IS2,Y’,C,L) at point in time t-1 of the reference neuron N(IS1,IS2,Y’,C,L) with the reference time-domain related index incremented with a shift amount equal to a base shift amount h of the window times the accumulated up / down- sampling factor S at the output of the layer L, which is the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y. It will be clear that this principle can be extended to an arbitrary number of dimensions. Also the principle is similarly applicable for spectral input signals in analogy to the way in which spectral audio data is processed in the embodiment described with reference to FIG. 2, 3 and 4.

Claims

CLAIMS1. A neural network processor for processing subsequent windows (W 1, W2, ...Wn) of a time dependent signal, each window comprising a plurality (ns) of samples of the time dependent signal for subsequent points in time in a timeinterval (T) and each window of samples being shifted in time relative to its immediately preceding window with a time-step (AT), which time-step is smaller than the time-interval (T), the time step corresponding to a base shift amount (h) defined bythe neural network processor being configured to execute a neural network with a plurality of neural network layers (L), each with a respective plurality of neurons (N(...,Y,...L)), addressable with at least a time domain related index (Y) , the neural network processor being configured to perform respective operations for respective windows of samples, a respective operation comprising: computing respective result signal values (R(...,Y,...,L)) of neurons(N(... , Y, ... ,L)) of a layer L, determining differential result signal values (D(...,Y,...,L)) of the neurons (N(... , Y, ... ,L)) of the layer (L), providing respective output signal values (O(...,Y,...,L)) for respective ones of the neurons (N(...,Y,...,L)) of the layer (L) of which the differential result signal values (D(...,Y,...,L)) exceed a threshold value; wherein the differential result signal value (D(...,Y,...,L)) of a particular neuron (N(...,Y,...,L)) is computed dependent on a value of a reference neuron in the same layer with reference time domain related index (Y as specified below, the differential result signal value (D(...,Y,...,L)) being equal to the difference between the result signal value (R(...,Y,...,L)) of the particular neuron (N(...,Y,...,L)) obtained in the respective operation n and the result signal value (R(...,Y’,...,L)) obtained in the respective operation for the preceding window of sample with a reference neuron (N(...,Y,...,L)) in the layer having a reference time domain related index Y’ equal to the time domain related index incremented with a shift amount equal to a base shift amount h of the windowtimes the accumulated up / down-sampling factor S at the output of the layer L, which is the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y.

2. The neural network processor according to claim 1, wherein the time dependent signal is a video signal or an audio signal.

3. The neural network processor according to claim 1, wherein the reference time domain related index Y’ is computed from the time domain related index Y of the particular neuron as:Y' = (Y + [h ■ SJ) mod sY where mod is the modulo operation, h is the base shift amount of the window, being h =and S is the accumulated up / down- sampling factor at the output of the layer L being the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y.

4. The neural network processor according to claim 2, wherein the reference time domain related index Y’ is computed from the time domain related index Y of the particular neuron as:Y' = (Y + [h ■ SJ) mod sY where mod is the modulo operation, h is the base shift amount of the window, being h =and S is the accumulated up / down- sampling factor at the output of the layer L being the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y.

5. The neural network processor according to claim 1, wherein each window (Wl, W2, ...Wn) is a spectrogram of an input signal.

6. The neural network processor according to claim 2, wherein each window (Wl, W2, ...Wn) is a spectrogram of an input signal.

7. The neural network processor according to claim 1, wherein the base shift amount is an integer power of 2.

8. The neural network processor according to claim 2, wherein the base shift amount is an integer power of 2.

9. The neural network processor according to claim 7, wherein the base shift amount is the integer power of 2 closest to the square root of the size of the first neural network layer in the direction of the time domain related index.

10. The neural network processor according to claim 8, wherein the base shift amount is the integer power of 2 closest to the square root of the size of the first neural network layer in the direction of the time domain related index.

11. The neural network processor according to claim 1, wherein a duration of the time window is at least five times the time shift value.

12. The neural network processor according to claim 2, wherein a duration of the time window is at least five times the time shift value.

13. A data processing system, comprising a Short-Time Fourier Transform (STFT) unit (2), a windowing unit (3) and a neural network processor (1) according to claim 1, wherein the STFT unit (2) receives at its input an input signal M(t) which is function of time and provides at its output a spectrogram (S(x,y)) as a function of time, wherein the windowing unit (3) repeatedly selects a window to be processed by the neural network processor (1), each window comprising a plurality of spectrograms obtained in a time-interval (T) and being shifted in time relative to its immediately preceding window with a time-step (AT), which time-step is smaller than the time-interval T.

14. A data processing system, comprising a Short-Time Fourier Transform (STFT) unit (2), a windowing unit (3) and a neural network processor (1) according to claim 2, wherein the STFT unit (2) receives at its input an input signal M(t) which is function of time and provides at its output a spectrogram (S(x,y)) as a function of time, wherein the windowing unit (3) repeatedly selects awindow to be processed by the neural network processor (1), each window comprising a plurality of spectrograms obtained in a time-interval (T) and being shifted in time relative to its immediately preceding window with a time-step (AT), which time-step is smaller than the time-interval T.

15. A neural network processing method for processing subsequent windows (Wl, W2, ...Wn) of a time dependent signal, each window comprising a plurality (ns) of samples of the time dependent signal for subsequent points in time in a time-interval (T) and each window of samples being shifted in time relative to its immediately preceding window with a time-step (AT), which time-step is smaller than the time-interval (T), the time step corresponding to a base shift amount (h) defined bythe neural network processing method comprising executing a neural network with a plurality of neural network layers (L), each with a respective plurality of neurons (N(...,Y,...L)), addressable with at least a time domain related index (Y), the neural network processor perform respective operations for respective windows of samples, a respective operation comprising: computing respective result signal values (R(...,Y,...,L)) of neurons(N(... , Y, ... ,L)) of a layer L, determining differential result signal values (D(...,Y,...,L)) of the neurons(N(... , Y, ... ,L)) of the layer (L), providing respective output signal values (O(...,Y,...,L)) for respective ones of the neurons (N(...,Y,...,L)) of the layer (L) of which the differential result signal values (D(...,Y,...,L)) exceed a threshold value; wherein the differential result signal value (D(...,Y,...,L)) of a particular neuron (N(...,Y,...,L)) is computed dependent on a value of a reference neuron in the same layer with reference time domain related index (Y as specified below, the differential result signal value (D(...,Y,...,L)) being equal to the difference between the result signal value (R(...,Y,...,L)) of the particular neuron (N(...,Y,...,L)) obtained in the respective operation n and the result signal value(R(...,Y’,...,L)) obtained in the respective operation for the preceding window of sample with a reference neuron (N(...,Y,...,L)) in the layer having a reference time domain related index Y’ equal to the time domain related index incremented with a shift amount equal to a base shift amount h of the window times the accumulated up / down-sampling factor S at the output of the layer L, which is the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y.

16. The neural network processing method according to claim 15, wherein the time dependent signal is a video signal or an audio signal.

17. The neural network processing method according to claim 15, further comprising receiving an input signal (M(t)) which is function of time and generating a spectrogram (S(x,y)) of the input signal as a function of time, repeatedly selecting a window comprising a plurality of spectrograms obtained in a time-interval (T) and being shifted in time relative to its immediately preceding window with a time-step (AT), which time-step is smaller than the time-interval T and subsequently processing the selected windows with the neural network processing method.

18. The neural network processing method according to claim 16, further comprising receiving an input signal (M(t)) which is function of time and generating a spectrogram (S(x,y)) of the input signal as a function of time, repeatedly selecting a window comprising a plurality of spectrograms obtained in a time-interval (T) and being shifted in time relative to its immediately preceding window with a time-step (AT), which time-step is smaller than the time-interval T and subsequently processing the selected windows with the neural network processing method.

19. A tangible computer-readable medium having computer-executable instructions stored thereon that, when executed by a processor, perform a method for neural network processing method for processing subsequent windows (W 1,W2, ...Wn) of a time dependent signal, each window comprising a plurality (ns) of samples of the time dependent signal for subsequent points in time in a timeinterval (T) and each window of samples being shifted in time relative to its immediately preceding window with a time-step (AT), which time-step is smaller than the time-interval (T), the time step corresponding to a base shift amount (h) defined bythe neural network processing method comprising executing a neural network with a plurality of neural network layers (L), each with a respective plurality of neurons (N(...,Y,...L)), addressable with at least a time domain related index (Y), the neural network processor perform respective operations for respective windows of samples, a respective operation comprising: computing respective result signal values (R(...,Y,...,L)) of neurons (N(... , Y, ... ,L)) of a layer L, determining differential result signal values (D(...,Y,...,L)) of the neurons (N(... , Y, ... ,L)) of the layer (L), providing respective output signal values (O(...,Y,...,L)) for respective ones of the neurons (N(...,Y,...,L)) of the layer (L) of which the differential result signal values (D(...,Y,...,L)) exceed a threshold value; wherein the differential result signal value (D(...,Y,...,L)) of a particular neuron (N(...,Y,...,L)) is computed dependent on a value of a reference neuron in the same layer with reference time domain related index (Y as specified below, the differential result signal value (D(...,Y,...,L)) being equal to the difference between the result signal value (R(...,Y,...,L)) of the particular neuron (N(...,Y,...,L)) obtained in the respective operation n and the result signal value (R(...,Y’,...,L)) obtained in the respective operation for the preceding window of sample with a reference neuron (N(...,Y,...,L)) in the layer having a reference time domain related index Y’ equal to the time domain related index incremented with a shift amount equal to a base shift amount h of the window times the accumulated up / down-sampling factor S at the output of the layer L,which is the ratio sY / ns, wherein sY is the size of the layer L in the direction of the time domain related axis Y.

20. The tangible computer readable medium according to claim 19, wherein the time dependent signal to be processed is a video signal or an audio signal.

Images