Apparatus for providing processed audio signal, apparatus for providing neural network parameters, method and computer program

Abstract

Embodiments according to the invention relate to an apparatus for providing a processed audio signal, an apparatus for providing neural network parameters, a method and a computer program. Embodiments in accordance with the present invention relate to improved normalized stream based speech enhancement for conditional input representation using all-pole gamma-maton filter banks. Embodiments in accordance with the present invention relate to improved normalized stream-based speech enhancement with different input conditions.

Description

Technical Field

[0001] Embodiments according to the invention relate to an apparatus for providing a processed audio signal, an apparatus for providing neural network parameters, a method and a computer program.

[0002] Embodiments according to the present invention relate to improved normalized stream based speech enhancement using an All-Pole Gammatone Filterbank for conditional input representation.

[0003] Embodiments according to the present invention relate to improved normalized stream based speech enhancement with different input conditions. Background Art

[0004] Speech enhancement (SE) aims to improve the quality of an impaired speech signal, for example in terms of intelligibility. The impairment may be caused, for example, by background noise.

[0005] Applications of speech enhancement include, among others, automatic speech recognition[2], speech coding[3], hearing assistance[4], and broadcasting[5].

[0006] Therefore, there are currently known numerous methods for speech enhancement that involve distinguishing a target speech signal from an interfering background.

[0007] Therefore, a wide range of methods have been developed, including more traditional techniques such as spectral subtraction [6], Wiener filters [7] or subspace methods [8], as well as methods based on Deep Neural Networks (DNNs).

[0008] For example, according to common approaches, a separation mask is estimated by minimizing a distance metric to extract clean speech components in the time-frequency (TF) domain [9, 10] or a learned subspace

[11] .

[0009] Nevertheless, in recent years, there has been a growing interest in generative methods that attempt to characterize the probability distribution of speech signals. The most prominent examples include Generative Adversarial Networks (GANs) [12, 13], Variational Autoencoders (VAEs)

[14] , autoregressive models

[15] , and diffusion probability models

[16] .

[0010] However, these conventional methods still have shortcomings in enhancing audio quality. For example, even the most advanced GAN-based methods still produce unsatisfactory results when perceptually evaluated through listening tests, especially at low signal-to-noise ratios.

[0011] In view of the above, it is desirable to create a concept for audio signal enhancement that provides an improved trade-off between computational complexity, training efficiency, and achievable audio quality.

[0012] This object is achieved by the subject matter of the independent claim. Further advantageous aspects are the subject matter of the dependent claims. Summary of the invention

[0013] In the following, embodiments according to the first aspect of the present invention are discussed. It should be noted that the presentation of embodiments according to different aspects is only for ease of understanding. In addition, it should be noted that the embodiments according to the first aspect may optionally include any features, functions and / or details of any embodiment of any other inventive aspects (particularly any embodiment of the second and / or third aspects) alone or in combination. Vice versa, the embodiments according to any other inventive aspects (particularly any embodiment of the second and / or third aspects) may optionally include any features, functions and / or details of any embodiment of the first aspect alone or in combination.

[0014] Furthermore, it is noted that features, functions and details in brackets, such as (features), are optional.

[0015] Embodiments according to a first aspect of the present invention include a method for providing a processed audio (e.g., speech) signal (e.g., an enhanced audio signal) (e.g., an enhanced speech signal or an enhanced generic audio signal; e.g., x^, e.g., a clean signal x extracted from the noisy speech signal y, e.g., y=x+n(noise, e.g., a noisy background)) based on an input audio (e.g., speech) signal (e.g., a distorted audio signal, e.g., a noisy speech signal y, e.g., a clean signal x extracted from the noisy speech signal y, e.g., y=x+n(noise, e.g., a noisy background)). ), wherein the device is configured to process (e.g., using affine scaling, or using a series of radial scaling operations) a noise signal (e.g., z) or a signal derived from the noise signal using one or more stream blocks (e.g., using a stream block system, e.g., including an affine coupling layer, e.g., including a reversible convolution) to obtain a processed (e.g., enhanced) audio signal (e.g., x^, e.g., ). The apparatus is configured to adapt processing performed using one or more stream blocks according to an input audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y; e.g. according to a noisy time-domain speech sample) and using a neural network (e.g. the neural network may provide one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream block based on the distorted audio signal and preferably also based on at least a portion of the noise signal or a processed version thereof).

[0016] Furthermore, the apparatus is configured to obtain a preprocessed representation of the input audio signal (e.g., a “conditioned signal representation”, wherein, for example, the input audio signal can be used as a conditioned signal for adapting processing performed using one or more stream blocks) using a filter bank comprising a time resolution and / or frequency resolution adapted to the time resolution and / or frequency resolution of the human auditory system (e.g., using an all-pole gammatone filter bank) (which can, for example, comprise a set of bandpass filters, wherein, for example, the bandpass filters can be infinite impulse response bandpass filters) (where, for example, the time / resolution and / or frequency resolution of the individual filters of the filter bank can be adapted to the time resolution and / or frequency resolution of the human auditory system).

[0017] Furthermore, the neural network is configured to receive a preprocessed representation of the input audio signal (eg, as an input value) (and, optionally, to provide one or more processing parameters to the stream block based on the preprocessed representation of the input audio signal).

[0018] Embodiments according to the first aspect of the invention are based on the discovery that, for example, for speech enhancement purposes, processing of an audio signal can be performed directly using stream block processing, which can, for example, model a generation process. It has been found that stream block processing allows processing of a noise signal, such as a noise signal z, such as a noise signal generated by or stored in the device, in a manner conditioned on an input audio signal (e.g. a noisy audio signal y). The noise signal z represents (or includes) a given (e.g. a simple or complex) probability distribution, preferably a Gaussian probability distribution. It has been found that when processing a noise signal conditioned on a distorted audio signal, an enhanced clean portion of the input audio signal is provided as a result of the processing, without introducing the clean portion (e.g. without introducing the noisy background) as input to the device.

[0019] Furthermore, the inventors have recognised that the use of a filter bank comprising a time resolution and / or a frequency resolution adapted to the time resolution and / or the frequency resolution of the human auditory system allows obtaining a preprocessed representation of the input audio signal with an improved audio quality, which can therefore improve the adaptation quality of the processing of the input audio signal, e.g. by a neural network in the form of a noise signal via a streaming block, and thus improve the quality of the processed audio signal. In particular, it has been recognised that by taking into account human hearing characteristics, the preprocessing of distorted (e.g. enhanced) audio signals in the context of noise shaping can be significantly improved. Furthermore, the inventors have found that by adapting the design of the respective filter bank in terms of time resolution and / or frequency resolution, human hearing characteristics can be taken into account.

[0020] Furthermore, it has been recognized that preprocessing using filter banks can provide particularly meaningful input information to the neural network, which means that the neural network can effectively determine the processing parameters of the stream block. It has been recognized that the output of the filter bank adapted to the human auditory system contains the most important information in a "condensed" form and is therefore very suitable as an input to the neural network. Therefore, it has been recognized that the filter bank is very suitable as an audio enhancement based on the preprocessing stream block.

[0021] Hence, an improved trade-off between computational complexity, training efficiency, and achievable audio quality can be achieved.

[0022] According to an embodiment of the first aspect of the invention, the time resolution of the filter bank and the frequency resolution of the filter bank approximate the time resolution and frequency resolution of the human auditory system. The inventors have recognized that using a perceptually motivated filter bank, such as imitating or approximating the human auditory system, allows improving the quality of the preprocessed representation of the input audio signal.

[0023] According to an embodiment of the first aspect of the invention, the filters of the filter bank (eg at least a plurality of filters in the filter bank, or even all filters in the filter bank) are infinite impulse response filters. Such filters can be efficiently implemented.

[0024] According to an embodiment of the first aspect of the present invention, the filter bank is an all-pole gammatone filter bank (eg a complex all-pole gammatone filter bank). The inventors found that the all-pole gammatone filter bank achieves a good trade-off in terms of computational complexity and representation of human hearing characteristics.

[0025] Embodiments according to a first aspect of the present invention include a method for providing a processed audio (e.g., speech) signal (e.g., an enhanced audio signal) (e.g., an enhanced speech signal or an enhanced generic audio signal; e.g., x^, e.g., a clean signal x extracted from the noisy speech signal y, e.g., y=x+n(noise, e.g., a noisy background)) based on an input audio (e.g., speech) signal (e.g., a distorted audio signal, e.g., a noisy speech signal y, e.g., a clean signal x extracted from the noisy speech signal y, e.g., y=x+n(noise, e.g., a noisy background)). ), wherein the device is configured to process (e.g., using affine scaling, or using a series of radial scaling operations) a noise signal (e.g., z) or a signal derived from the noise signal using one or more stream blocks (e.g., using a stream block system, e.g., including an affine coupling layer, e.g., including a reversible convolution) to obtain a processed (e.g., enhanced) audio signal (e.g., x^, e.g. ). Furthermore, the apparatus is configured to adapt the processing performed using one or more stream blocks according to the input audio signal (e.g. a distorted audio signal, e.g. a noisy speech signal y; e.g. a noisy time domain speech sample) and using a neural network (the neural network optionally providing one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream block based on the distorted audio signal and preferably also based on at least a portion of the noise signal or a processed version thereof).

[0026] In addition, the apparatus is configured to obtain a preprocessed representation (e.g., a “conditioned signal representation,” where, for example, the input audio signal can be used as a conditioned signal for adapting processing performed using one or more stream blocks) of the input audio signal using an all-pole gammatone filter bank (which can, for example, include a set of bandpass filters, where, for example, the bandpass filters can be infinite impulse response bandpass filters), and the neural network is configured to receive the preprocessed representation of the input audio signal (e.g., as an input value) (and provide one or more processing parameters to the stream block based on the preprocessed representation of the input audio signal).

[0027] According to an embodiment of the first aspect of the invention, the all-pole gammatone filter bank is configured to obtain a plurality of channel signals associated with a plurality of (overlapping or non-overlapping) frequency bands (wherein, for example, the ratio between the width of the lowest frequency band of the all-pole gammatone filter bank and the width of the highest frequency band of the all-pole gammatone filter bank is in the range between 1:10 and 1:50), wherein the width of the frequency bands increases monotonically with increasing center frequencies of the respective frequency bands, and / or wherein the width of the frequency bands is adapted according to a psychoacoustic model (for example, according to the Bark scale or the ERB scale), and / or wherein the center frequencies of the frequency bands are adapted according to a psychoacoustic model (for example, according to the Bark scale or the ERB scale). Preprocessing the plurality of channel signals of different frequency bands, in particular according to a psychoacoustic model, allows a good trade-off in terms of computational complexity and processed signal quality.

[0028] According to an embodiment of the first aspect of the invention, an all-pole gammatone filter bank comprises a plurality of filters (e.g., infinite impulse response filters), the center frequencies of the filters comprising a constant distance on the Bark scale (e.g., optionally, within a tolerance of + / -10% or + / -5%), and the bandwidth also increases (e.g., proportional to the Bark bandwidth) as the frequency increases. The inventors have recognized that filter design taking into account the Bark scale can allow for effective representation of human hearing characteristics in the filtering step.

[0029] According to an embodiment of the first aspect of the invention, the all-pole gammatone filter bank is configured to at least partially compensate for different group delays between different filters (e.g., using the look-ahead of one or more filters of the all-pole gammatone filter bank) (where, for example, the look-ahead of different filters is configured according to or in accordance with the group delays at the center frequencies of different filters of the all-pole gammatone filter bank). The inventors have recognized that compensating for group delays can improve filter efficiency.

[0030] According to an embodiment of the first aspect of the invention, the transfer function of the all-pole gammatone filter bank does not include any finite zeros.

[0031] According to an embodiment of the first aspect of the invention, the imaginary parts of the poles of the transfer function of the all-pole gammatone filter bank all include the same sign (e.g., all include a positive sign, or alternatively, all include a negative sign). It has been found that such a selection of the transfer function poles results in the output signal of the filter bank being effectively used by the neural network.

[0032] According to an embodiment of the first aspect of the invention, the all-pole gammatone filter bank is a complex all-pole gammatone filter bank (eg an all-pole gammatone filter bank providing a complex-valued output signal in response to a real-valued input signal).

[0033] According to an embodiment of the first aspect of the invention, one or more poles of the transfer function of the all-pole gammatone filter bank (or even all poles of the transfer function of the all-pole gammatone filter bank) coincide.

[0034] The inventors have realised that the implementation variants of the above described all-pole gammatone filter bank allow providing an efficient filter bank for pre-processing of an input audio signal to provide an input signal to a neural network, thereby allowing providing well-adapted parameters for the processing in the stream block.

[0035] According to an embodiment of the first aspect of the invention, the apparatus is configured to obtain a preprocessed representation of the input audio signal based on the amplitude of the (e.g. complex-valued) output signal of the all-pole gammatone filter bank (wherein, for example, the apparatus is configured to determine the amplitude of the complex-valued output value of the all-pole gammatone filter bank). Additionally or alternatively, the apparatus is configured to ignore phase information of the (e.g. complex-valued) output signal of the all-pole gammatone filter bank. Thus, in order to improve processing efficiency, the amount of information processed can be reduced.

[0036] According to an embodiment of the first aspect of the invention, an all-pole gammatone filter bank is configured to provide 20 to 100 output signals (e.g., output signals associated with 20 to 100 different frequency bands), wherein the output signals of the filter bank can, for example, be input into a neural network. It has been recognized that such a number of output signals of the filter bank is well managed by the neural network and well reflects the most psychoacoustically relevant features of the input audio signal. Therefore, the neural network can effectively provide good processing parameters for the stream block based on such a set of input signals.

[0037] According to an embodiment of the first aspect of the invention, the apparatus is configured to apply a plurality of convolutions to a set of output values ​​of an all-pole gammatone filter bank (e.g., to a set of output values ​​of an all-pole gammatone filter bank associated with a plurality of frequency bands and with a plurality of time instances) or to a set of amplitude values ​​derived from the output values ​​of the all-pole gammatone filter bank (e.g., to amplitude values ​​derived from a set of output values ​​of an all-pole gammatone filter bank associated with a plurality of frequency bands and with a plurality of time instances) to obtain input values ​​of the neural network. Thus, in addition to the filter bank, a pre-processing unit configured to perform convolution, such as an additional pre-processing unit, may be implemented between the filter bank and the neural network. The inventors have recognized that the use of a combination of filtering and subsequent convolution may improve the input signal of the neural network.

[0038] According to an embodiment of the first aspect of the present invention, the apparatus is configured to apply a depthwise separable convolution (e.g., a plurality of depthwise separable convolutions) to a set of output values ​​of an all-pole gammatone filter bank (e.g., to a set of output values ​​of an all-pole gammatone filter bank associated with a plurality of frequency bands and with a plurality of time instances), or to a set of amplitude values ​​derived from the output values ​​of an all-pole gammatone filter bank (e.g., to amplitude values ​​derived from a set of output values ​​of an all-pole gammatone filter bank associated with a plurality of frequency bands and with a plurality of time instances), to obtain input values ​​of a neural network (wherein the input values ​​of the neural network, which may, for example, represent a preprocessed version of an input audio signal, may, for example, include (for each input audio signal sample) a plurality of convolution values, where the convolution values ​​are, for example, the result of different convolutions of a representation of the input audio signal with different convolution kernels). This may allow a reduction in the complexity and / or number of parameters involved in the processing, such as Figure 4 shown.

[0039] In addition, according to some embodiments of the first aspect of the invention, the use of a filter bank can improve the result, for example in terms of audio quality, but may increase the complexity of processing, for example due to an increase in the number of parameters. However, based on the significant improvement in processed (e.g., enhanced) audio quality and the relatively small increase in the number of parameters, a better trade-off between audio quality and processing complexity can also be achieved.

[0040] In the following embodiments, the second aspect of the present invention is discussed. It should be noted that the embodiments according to the second aspect may optionally include any features, functions and / or details of any embodiment of any other inventive aspect (particularly any embodiment of the first and / or third aspect) alone or in combination. Vice versa, the embodiments according to any other inventive aspect (particularly any embodiment of the first and / or third aspect) may optionally include any features, functions and / or details of any embodiment of the second aspect alone or in combination.

[0041] Embodiments according to the second aspect of the present invention include a method for providing a processed audio (e.g., speech) signal (e.g., an enhanced audio signal) (e.g., an enhanced speech signal or an enhanced generic audio signal; e.g., x^, e.g., a clean signal x extracted from the noisy speech signal y) based on an input audio (e.g., speech) signal (e.g., a distorted audio signal, e.g., a noisy speech signal y, e.g., a clean signal x extracted from the noisy speech signal y, y=x+n(noise, e.g., a noisy background)). ), wherein the device is configured to process (e.g., using affine scaling, or using a series of affine scaling operations) a noise signal (e.g., z) or a signal derived from the noise signal using one or more stream blocks (e.g., using a stream block system, e.g., including an affine coupling layer, e.g., including a reversible convolution) to obtain a processed (e.g., enhanced) audio signal (e.g., x^, e.g. ).

[0042] Furthermore, the apparatus is configured to adapt the processing performed using one or more stream blocks based on an input audio signal (e.g. a distorted audio signal, e.g. based on a noisy speech signal y; e.g. based on a noisy time domain speech sample) and using a neural network (which optionally provides one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream block based on the distorted audio signal and preferably also based on at least a portion of the noise signal or a processed version thereof).

[0043] Furthermore, the apparatus is configured to apply a depthwise separable convolution to a representation of the input audio signal to obtain a preprocessed representation of the input audio signal (wherein the preprocessed version of the input audio signal may, for example, comprise (for each input audio signal sample) a plurality of convolution values, where the convolution values ​​are, for example, results of different convolutions of the representation of the input audio signal with different convolution kernels).

[0044] Furthermore, the neural network is configured to receive a pre-processed representation of the input audio signal (and to provide one or more processing parameters to the stream block based on the pre-processed representation of the input audio signal).

[0045] Embodiments according to the second aspect are based on the finding that based on preprocessing of the input audio signal the quality of the information provided to the neural network can be improved. Based on the preprocessing the neural network can therefore provide adaptation information to the stream block, wherein noise shaping can be adapted or guided based on the input audio signal. The inventors have also recognised that an implementation of such preprocessing (e.g. optionally with or without a previous filtering step) (e.g. using a filter bank) can be efficiently performed based on depthwise separable convolutions. In this way the complexity of the processing of the input audio signal can be reduced by reducing the number of parameters involved, see Figure 4 . It has been recognized that storing the parameters of depthwise separable convolutions consumes significantly less storage space than storing the parameters of non-depthwise separable convolutions. In addition, the application of depthwise separable convolutions can also significantly reduce the workload compared to non-depthwise separable convolutions, because the processing blocks can be reused multiple times without updating the parameters. Furthermore, it has been found that the use of depthwise separable convolutions does not significantly weaken the processing results. Therefore, the use of depthwise separable convolutions brings a good trade-off between processing complexity and achievable audio quality.

[0046] According to an embodiment of the second aspect of the present invention, the depthwise separable convolution is configured to perform convolution in time and convolution in frequency.

[0047] According to an embodiment of the second aspect of the present invention, the apparatus is configured to obtain a representation of the input audio signal using an all-pole gammatone filter bank, and the apparatus is configured to apply a depthwise separable convolution to the representation of the input audio signal obtained using the all-pole gammatone filter bank.

[0048] According to an embodiment of the second aspect of the invention, the representation of the input audio signal obtained using an all-pole gammatone filter bank comprises a plurality (e.g. between 20 and 100) of sub-band signals (e.g. a plurality of signal values ​​for each input audio signal sample, e.g. between 20 and 100).

[0049] According to an embodiment of the second aspect of the invention, the apparatus is configured to apply different convolutions to a plurality of sub-band signals to obtain an input signal to a neural network (where, for example, the number of input signals is larger than the number of sub-band signals provided by an all-pole gammatone filter bank, for example at least ten times larger).

[0050] According to an embodiment of the second aspect of the invention, the apparatus is configured to apply separate time convolutions (e.g. 80 time convolutions, each convolution considering 3 values ​​associated with different time instances) to a plurality of signals representing an input audio signal (e.g. channel signals applied to an all-pole gammatone filter bank; e.g. 80 channel signals applied to an all-pole gammatone filter bank) to obtain a plurality of time convolved signal values ​​(wherein the time convolutions may, for example, be defined by different time convolution kernels, e.g. associated with different frequencies or with different frequency bands) (e.g. 80 result values ​​of these 80 separate time convolutions).

[0051] In addition, the apparatus is configured to apply multiple convolutions in frequency (e.g., 2000 different convolutions) to a given set of temporal convolution signal values ​​(e.g., to a given set of convolution signal values ​​associated with a given time instance) (e.g., to 80 result values ​​of 80 separate temporal convolutions) to obtain multiple input values ​​for the neural network.

[0052] Optionally, as an example, each input value of the neural network can be obtained using a corresponding convolution, and as an example, different input values ​​of the neural network (e.g., associated with the same time) can be obtained using corresponding convolutions based on a common (e.g., given) set of time convolution signal values.

[0053] Furthermore, optionally, as an example, the number of input values ​​of the neural network (e.g., per sample time instance) may be larger than the number of representative values ​​of the input audio signal to which the depthwise separable convolution is applied, e.g., at least 10 times larger.

[0054] According to an embodiment of the second aspect of the present invention, the device is configured to apply a depthwise separable convolution to a representation of an input audio signal to map an input space (e.g., defined by values ​​of channel signals associated with different frequency bands and obtained from the input audio signal, e.g., using an all-pole gammatone filter bank) to a higher dimension (or multiple higher dimensions).

[0055] According to an embodiment of the second aspect of the present invention, the apparatus is configured to perform multiple convolutions in frequency (e.g. 2000 convolutions in frequency) based on the same set of result values ​​of (e.g. 80) respective time convolutions, wherein the respective time convolutions are performed respectively on signals of a frequency domain representation of the input audio signal (e.g. provided by an all-pole gammatone filter bank).

[0056] According to an embodiment of the second aspect of the present invention, one or more stream blocks include at least one dual-coupled stream block (for example, within an affine coupling layer), wherein the (corresponding) dual-coupled stream block is configured to apply a first affine transformation (whose transformation coefficients can be determined, for example, using a neural network, for example, using a subnetwork providing s1 (for example, s1) and t1 (for example, t1)) to a first part (for example, x1, for example, x1) of an input signal (for example, x) to be modified by the (for example, the corresponding) dual-coupled stream block, and wherein the (for example, the corresponding) dual-coupled stream block is configured to apply a second affine transformation (for example, whose transformation coefficients can be determined, for example, using a neural subnetwork providing s2 (for example, s2) and t2 (for example, t2)) to a second part (which can be different from the first part) (for example, x2, for example, x2) of the input signal (for example, x) to be modified by the (corresponding) dual-coupled stream block.

[0057] According to an embodiment of the second aspect of the invention, the apparatus is configured to obtain a preprocessed representation (e.g., a “conditioned signal representation”) of an input audio signal using a filter bank (e.g., using an all-pole gammatone filter bank), wherein, for example, the input audio signal can be used as a conditioned signal for adapting processing performed using one or more stream blocks, the filter bank comprising a time resolution and / or frequency resolution adapted to the time resolution and / or frequency resolution of the human auditory system (the filter bank may, for example, comprise a set of bandpass filters, wherein, for example, the bandpass filters may be infinite impulse response bandpass filters) (wherein, for example, the time / resolution and / or frequency resolution of the individual filters of the filter bank may be adapted to the time resolution and / or frequency resolution of the human auditory system. In addition, the neural network is configured to receive the preprocessed representation of the input audio signal (e.g., as an input value) (and, optionally, to provide one or more processing parameters to the stream block based on the preprocessed representation of the input audio signal).

[0058] According to an embodiment of the second aspect of the invention, the apparatus is configured to obtain a preprocessed representation (e.g., a "conditioned signal representation", where, for example, the input audio signal can be used as a conditioned signal for adapting processing performed using one or more stream blocks) of an input audio signal using an all-pole gamma filter bank (which can, for example, include a set of bandpass filters, where, for example, the bandpass filters can be infinite impulse response bandpass filters). In addition, the neural network is configured to receive the preprocessed representation of the input audio signal (e.g., as an input value) (and is configured to provide one or more processing parameters to the stream block based on the preprocessed representation of the input audio signal).

[0059] In the following embodiments, the third aspect of the present invention is discussed. It should be noted that the embodiments according to the third aspect may optionally include any features, functions and / or details of any embodiment of any other inventive aspects (particularly any embodiment of the first and / or second aspects) alone or in combination. Vice versa, the embodiments according to any other inventive aspects (particularly any embodiment of the first and / or second aspects) may optionally include any features, functions and / or details of any embodiment of the third aspect alone or in combination.

[0060] Embodiments according to the third aspect of the present invention include a method for providing a processed audio (e.g., speech) signal (e.g., an enhanced audio signal) (e.g., an enhanced speech signal or an enhanced generic audio signal; e.g., x^, e.g., a clean signal x extracted from the noisy speech signal y, y=x+n(noise, e.g., a noisy background)) based on an input audio (e.g., speech) signal (e.g., a distorted audio signal, e.g., a noisy speech signal y, e.g., a clean signal x extracted from the noisy speech signal y, y=x+n(noise, e.g., a noisy background)). ), wherein the device is configured to use one or more stream blocks (e.g., using a stream block system, e.g., including an affine coupling layer, e.g., including a reversible convolution) to process (e.g., using affine scaling, or using a series of affine scaling operations) a noise signal (e.g., z) or a signal derived from the noise signal to obtain a processed (e.g., enhanced) audio signal (e.g., x^, e.g. ).

[0061] Furthermore, the apparatus is configured to adapt the processing performed using one or more stream blocks based on an input audio signal (e.g. a distorted audio signal, e.g. a noisy speech signal y; e.g. a noisy time domain speech sample) and using a neural network (which optionally provides one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream block based on the distorted audio signal and preferably also based on at least a portion of the noise signal or a processed version thereof). The one or more stream blocks include at least one dual-coupled stream block (e.g. within an affine coupling layer), wherein the (corresponding) dual-coupled stream block is configured to apply a first affine transformation (whose transformation coefficients may be determined, for example, using a neural network, e.g. using a subnetwork providing s1 (e.g. s1) and t1 (e.g. t1)) to a first part (e.g. x1, e.g. x1) of an input signal (e.g. x) to be modified by the (corresponding) dual-coupled stream block, and wherein the (corresponding) dual-coupled stream block is configured to apply a second affine transformation (whose transformation coefficients may be determined, for example, using a neural subnetwork providing s2 (e.g. s2) and t2 (e.g. t2)) to a second part (which may be different from the first part) (e.g. x2, e.g. x2) of the input signal (e.g. x) to be modified by the (corresponding) dual-coupled stream block.

[0062] The inventors recognize that the use of the dual coupling scheme of the present invention allows the entire input signal to be directly processed, thereby allowing better overall performance. Therefore, expressibility (e.g., expressiveness) can be improved. It has been found that by using two separate affine transformations (which typically include some independence), the generation of an output audio signal can be facilitated. For example, it can be easily understood that dividing a signal into two components, processing the components separately, and recombining the processed components can reduce the statistical dependence of the samples that make up the components. This helps to convert the input signal into white noise. Now, it can be understood that, in another direction, the processing using two (e.g., separate) affine transformations is very suitable for converting a noise signal into a desired processed audio signal. In other words, it has been found that dividing a signal into two (or more) parts, applying separate affine transformations to these two (or more) parts (wherein the affine transformations are controlled by a neural network), and recombining the affine transformed parts are very suitable for generating a high-quality output signal based on a noise signal.

[0063] As an example, a flow block may include an affine coupling layer (optionally together with a reversible convolution layer). In a first step, for example, the input signal may be subsampled. For example, using the optional reversible convolution layer, the input may be split, for example, in two halves, for example, where one part is provided to a subnetwork of a neural network, for example within a coupling layer, for example to learn affine transformation parameters of the second half. For example, the transformed signal may be concatenated with the unchanged second part and may, for example, be used as input to the next block. For example, the operation may be reversible, ensuring that the network is overall reversible, for example, although the subnetwork within the coupling layer that estimates the affine parameters does not need to be reversible. Fig.15 An example of a corresponding stream block with a neural network is shown in .

[0064] According to an embodiment of the third aspect, the apparatus is configured to adapt a first affine transformation (e.g., by receiving x1 and providing x2) to be performed by a dual coupled flow block using a neural network (e.g., using a first neural network or using a first subnet) according to a second affine transformed input signal (e.g., x2) affine transformation of the first affine transformation) and the apparatus is configured to modify the first part (e.g., x1) of the input signal (e.g., x) to be modified by the dual coupled stream block (e.g., according to the output signal of the first affine transformation, e.g., according to ) uses a neural network (e.g., uses a second neural network or uses a second subnet) to adapt to a second affine transformation to be performed by the dual coupled flow block (e.g., by receiving x2 and providing Affine transformation of .

[0065] According to an embodiment of the third aspect, the device is configured to apply processing using a sequence of multiple dual-coupled stream blocks, and the device is configured to apply reversible convolution (which can be trained, for example, in a training phase) to obtain an input signal of a second dual-coupled stream block based on an output signal of a previous first dual-coupled stream block.

[0066] According to an embodiment of the third aspect, the (corresponding) dual-coupled stream block is configured to separate an input signal (e.g., x) of the dual-coupled stream block to obtain a first part of the input signal (e.g., x1, e.g., x1) and a second part of the input signal (e.g., x2, e.g., x2), and apply respective affine transformations to the first part of the input signal and the second part of the input signal.

[0067] According to an embodiment of the third aspect, the apparatus is configured to connect the output signal of the first affine transformation and the output signal of the second affine transformation (e.g. and ) to obtain the output signal of the (corresponding) dual-coupled stream block (wherein the output signal of the corresponding dual-coupled stream block can be used as the input signal of a subsequent dual-coupled stream block (e.g. after a reversible 1x1 convolution)).

[0068] According to an embodiment of the third aspect, the device is configured to use a second part of the input signal (e.g. x2, e.g. x2) as an input signal of a neural network (e.g. a first neural network) for determining transformation parameters of a first affine transformation (wherein the (e.g. first) neural network (e.g. a subnetwork or a subnetwork of the neural network) also receives a representation of the input audio signal) and as an input signal of a second affine transformation (e.g. as an output signal to be affine transformed).

[0069] According to an embodiment of the third aspect, the apparatus is configured to use an output signal (e.g. ) as an input signal to a neural network (e.g. a second neural network) for determining transformation parameters of a second affine transformation (wherein the (e.g. second) neural network optionally also receives a representation of the input audio signal).

[0070] According to an embodiment of the third aspect, the dual-coupled stream block is configured to divide the input signal (e.g., x) to be modified by the (corresponding) dual-coupled stream block into two halves (e.g., into a first part or a first half x1, e.g., x1, and a second part of a second half x2, e.g., x2). In addition, the dual-coupled stream block is configured to use the second half (e.g., x2, e.g., x2) of the input signal (e.g., x) to be modified by the (corresponding) dual-coupled stream block to estimate (e.g., using a first neural network) parameters of an affine transformation, which will be applied to the first half (e.g., x1, e.g., x1) of the input signal (e.g., x) to be modified by the (corresponding) dual-coupled stream block.

[0071] According to an embodiment of the third aspect, the dual coupled stream block is configured to modify only the signal of the first part (e.g. x1, e.g. x1) of the input signal (e.g. x) to be modified by the (corresponding) dual coupled stream block in the first affine transformation (while the signal of the second part (e.g. x2, e.g. x2) is unchanged by the first affine transformation), and to modify only the signal of the first part (e.g. x1, e.g. x1) of the input signal (e.g. x) to be modified by the (corresponding) dual coupled stream block in the second affine transformation (while the signal of the first part (e.g. x1 and ) is maintained unchanged by the second affine transformation) in which the signal of the second part (e.g., x2, e.g., x2) of the modified input signal (e.g., x) is transformed.

[0072] In the following, further embodiments are discussed, in particular embodiments according to any of the first, second and / or third aspects of the present invention. It should be noted that the following embodiments may optionally include, alone or in combination, any features, functions and / or details of any of the embodiments of any other inventive aspects (in particular any of the first, second and / or third aspects). Vice versa, embodiments according to any of the embodiments of any other inventive aspects (in particular any of the first, second and / or third aspects) may optionally include, alone or in combination, any features, functions and / or details of any of the following embodiments.

[0073] According to an embodiment of the present invention, an input audio signal is represented by a set of time-domain audio samples (e.g., noisy time-domain audio (e.g., speech) samples, such as time-domain speech utterances) (wherein, for example, the time-domain audio samples of the input audio signal, or the time-domain audio samples derived therefrom, are input into a neural network) (wherein, for example, the time-domain audio samples of the input audio signal. Or, as an example, the time-domain audio samples derived therefrom, are processed in the neural network in the form of a time-domain representation without applying a transform to a transform domain representation (e.g., a spectral domain representation)).

[0074] According to an embodiment of the present invention, a neural network associated with a given stream block (e.g., a given stage of affine processing) of one or more stream blocks is configured to determine one or more processing parameters (e.g., s, t) of the given stream block based on a noise signal (z) or a signal derived from the noise signal and based on an input audio signal (y). It should be noted that the apparatus of the present invention may include multiple stream blocks, for example, each stream block has a corresponding neural network. However, the apparatus may also include a neural network having multiple sub-networks, wherein a set of sub-networks of the multiple sub-networks, for example, a set of 2 sub-networks, may be associated with a corresponding stream block.

[0075] According to an embodiment of the invention, a neural network associated with a given stream block (e.g., a given stage of affine processing) is configured to provide one or more parameters (e.g., s, t) of an affine processing (e.g., in an affine coupling layer) that is applied to a noise signal in the process, or to a processed version of a noise signal in the process, or to a portion of a noise signal in the process, or to a portion of a processed version of a noise signal in the process (e.g., z). The inventors recognized that adaptation of noise shaping using an affine transformation can be efficiently performed using a neural network that provides parameters of a corresponding affine transformation.

[0076] According to an embodiment of the present invention, a neural network associated with a given stream block (e.g., a given stage of affine processing) is configured to determine one or more parameters (e.g., s, t) of the affine processing based on a first portion (z1, e.g., z1) of the stream block input signal (z) and based on the input audio signal (y). In addition, the affine processing associated with the given stream block (e.g., a given stage of affine processing) is configured to apply the determined parameters (e.g., s, t) to a second portion (z2, e.g., z2) of the stream block input signal (z) to obtain an affine processed signal (z2^, e.g., ). In addition, the first part (z1, such as z1) of the input signal (z) of the flow block (which is not modified by the affine processing, for example) and the affine processed signal (z2^, such as ) forms (e.g. constitutes) the stream block output signal (z) of a given stream block (a given stage of affine processing) new ) (e.g., phase output signal). Therefore, embodiments according to the present invention may include a single coupling scheme, such as Fig.21 shown.

[0077] According to an embodiment of the present invention, the apparatus is configured to apply a reversible convolution (e.g., a 1x1 reversible convolution) to a stream block output signal (z ) of a given stream block (e.g., a given stage of affine processing). new )(stage output signal)(for example, for other subsequent stages after the first stage, it can be the input signal of the subsequent stage) to obtain the processed stream block output signal (z' new )(a processed version of the stream block output signal, such as a convolved version of the stream block output signal).

[0078] According to an embodiment of the invention, the apparatus is configured to apply a nonlinear expansion (eg an inverse μ-law transform, eg a restored μ-law transform) to the processed (eg enhanced) audio signal.

[0079] According to an embodiment of the present invention, the apparatus is configured to apply an inverse μ-law transform (e.g., an inverse μ-law function; e.g., by restoring the μ-law transform) as a nonlinear expansion to the processed (e.g., enhanced) audio signal (x^, e.g. ).

[0080] According to an embodiment of the present invention, the device is configured to The transformation is applied to the processed (eg enhanced) audio signal (x^, e.g. ), where sgn() is the symbolic function and μ is a parameter defining the level of unrolling.

[0081] According to an embodiment of the present invention, a training audio signal or a processed version thereof is processed in one or more training stream blocks to obtain (e.g., predetermined, e.g., stored in a device, e.g., stored in a remote server) neural network parameters of a neural network for processing a noise signal or a signal derived from the noise signal to obtain a training result signal, wherein the processing of the training audio signal or the processed version thereof using one or more training stream blocks is based on a distorted version of the training audio signal and adapted using a neural network.

[0082] In addition, neural network parameters of the neural network are determined so that characteristics (e.g., probability distribution) of the training result audio signal approximate or include predetermined characteristics (e.g., noise-like characteristics; e.g., Gaussian distribution). Optionally, the one or more neural networks used to provide the processed audio signal may be the same as the one or more neural networks used to provide the training result signal, wherein the training stream block may perform affine processing that is opposite to the affine processing performed in providing the processed audio signal).

[0083] In general, it is to be noted that embodiments according to the present invention may allow the use of a structurally similar or even identical device comprising a stream block and a neural network for training the device in the form of neural network training, as well as audio signal enhancement. Between training and signal enhancement, the affine transformation performed in the stream block may be inverted. Thus, for training based on the guidance of a clean speech signal by a distorted version thereof, the neural network may be trained to adapt the processing using the affine transformation in the stream block to obtain a training result audio signal having specific characteristics. In order to apply signal enhancement to the real input audio signal, the affine transformation may be inverted, and the device may be provided with a noise signal, such as a noise signal having specific characteristics, which may be shaped into an enhanced version of the real input audio signal based on the real input audio signal provided to the neural network to adapt to the inverted affine transformation.

[0084] According to an embodiment of the present invention, the device is configured to provide neural network parameters of a neural network for processing a noise signal or a signal derived from a noise signal. The device is configured to process a training audio signal or a processed version thereof using one or more stream blocks to obtain a training result signal, and the device is configured to adapt the processing of the training audio signal or a processed version thereof performed using one or more stream blocks based on a distorted version of the training audio signal and using a neural network. In addition, the device is configured to determine the neural network parameters of the neural network (e.g., using an evaluation of a cost function, such as an optimization function; for example, using a parameter optimization procedure) so that the characteristics (e.g., probability distribution) of the training result audio signal approximate or include predetermined characteristics (e.g., noise-like characteristics; for example, Gaussian distribution).

[0085] According to an embodiment of the present invention, the device includes a device for providing neural network parameters, wherein the device for providing neural network parameters is configured to provide neural network parameters of a neural network for processing a noise signal or a signal derived from a noise signal. In addition, the device for providing neural network parameters is configured to process a training audio signal or a processed version thereof using one or more training stream blocks to obtain a training result signal, and the device for providing neural network parameters is configured to adapt the processing of the training audio signal or the processed version thereof performed using one or more stream blocks based on a distorted version of the training audio signal and using a neural network. In addition, the device is configured to determine the neural network parameters of the neural network (e.g., using an evaluation of a cost function, such as an optimization function; for example, using a parameter optimization procedure) so that the characteristics (e.g., probability distribution) of the training result audio signal approximate or include predetermined characteristics (e.g., noise-like characteristics; for example, Gaussian distribution).

[0086] According to an embodiment of the present invention, one or more stream blocks are configured to synthesize a processed audio (eg speech) signal based on a noise signal under the guidance of an input audio (eg speech) signal.

[0087] According to an embodiment of the present invention, one or more stream blocks are configured to synthesize a processed audio (e.g., speech) signal based on a noise signal under the guidance of an input audio (e.g., speech) signal, using affine processing of sample values ​​of the noise signal or affine processing of sample values ​​of a signal derived from the noise signal, and processing parameters (e.g., s, t) of the affine processing are determined based on (e.g., time domain) sample values ​​of the input audio signal using a neural network.

[0088] According to an embodiment of the invention, the apparatus is configured to perform normalized stream processing to derive a processed audio signal from a noise signal (eg under the guidance of an input audio signal).

[0089] In the following, embodiments according to the first, second and third aspects, as well as embodiments of further and / or additional aspects are disclosed, for example, in particular. It should be noted that features, functions and / or details may be optionally combined with any other aspects, and features of other aspects may be optionally introduced into these aspects alone or in combination.

[0090] According to an embodiment of the present invention, a device is provided for providing neural network parameters for audio (e.g., speech) processing (such as, for example, edge weights (θ) of a neural network that provides a scaling factor (s) and an offset value (t) based on a portion of a clean audio signal (e.g., x1, e.g., x1) or a processed version thereof and based on a distorted audio signal (y) in a training mode, wherein the edge weights (θ) may correspond to edge weights of a neural network that provides a scaling factor (s) and an offset value (t) based on a portion of a noise signal (z) or a processed version thereof and based on an input audio signal (y) in an inference mode), wherein the device is configured to process a training audio (e.g., speech) signal (e.g., x) or a processed version thereof using one or more stream blocks (e.g., using a stream block system, e.g., including affine coupling layers, e.g., including reversible convolutions) (e.g., in multiple iterations) to obtain a training result signal (e.g., which should be equal to the noise signal). Furthermore, the apparatus is configured to adapt the processing performed using one or more stream blocks according to a distorted version (e.g. y) of the training audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y) and using a neural network (which is optionally based on the distorted version of the training audio signal and preferably also provides one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream block according to at least a portion of the training audio signal or a processed version thereof. Furthermore, the apparatus is configured to determine neural network parameters of the one or more neural networks (e.g. using an evaluation of a cost function, e.g. an optimization function; e.g. using a parameter optimization procedure) such that a feature (e.g. a probability distribution) of the training result audio signal approximates or includes a predetermined feature (e.g. a noise-like feature; e.g. a Gaussian distribution). In addition, the apparatus is configured to apply a depthwise separable convolution to a representation of a distorted version of a training audio signal (e.g., which may replace an input audio signal during training) to obtain a preprocessed representation of a distorted version of the training audio signal (wherein the preprocessed distorted version of the training audio signal may, for example, include (for each sample of the distorted version of the training audio signal) a plurality of convolution values, where the convolution values ​​are, for example, the result of different convolutions of the representation of the distorted training audio signal with different convolution kernels). In addition, the neural network is configured to receive the preprocessed representation of the distorted version of the training audio signal (and optionally to provide one or more processing parameters to the stream block based on the preprocessed representation of the distorted version of the training audio signal).

[0091] According to an embodiment of the present invention, a device is provided for providing neural network parameters for audio (e.g., speech) processing (such as, for example, edge weights (θ) of a neural network that provides a scaling factor (s) and an offset value (t) based on a portion of a clean audio signal (e.g., x1, e.g., x1) or a processed version thereof, and based on a distorted audio signal (y) in a training mode, and the edge weights (θ) may correspond to the edge weights of a neural network that provides a scaling factor (s) and an offset value (t) based on a portion of a noise signal (z) or a processed version thereof and based on an input audio signal (y) in an inference mode), wherein the device is configured to (e.g., in multiple iterations) process a training audio (e.g., speech) signal (e.g., x) or a processed version thereof using one or more stream blocks (e.g., using a stream block system, e.g., including an affine coupling layer, e.g., including a reversible convolution) to obtain a training result signal (which should be equal to the noise signal). Furthermore, the apparatus is configured to adapt processing performed using one or more stream blocks according to a distorted version (e.g. y) of a training audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y) and using a neural network (which is optionally based on a distorted version of the training audio signal and preferably also provides one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream block based on at least a portion of the training audio signal or a processed version thereof).

[0092] In addition, the device is configured to determine neural network parameters of the neural network (e.g., using an evaluation of a cost function, such as an optimization function; for example, using a parameter optimization procedure) so that characteristics (e.g., probability distribution) of the training result audio signal approximate or include predetermined characteristics (e.g., noise-like characteristics; such as a Gaussian distribution).

[0093] In addition, one or more stream blocks include at least one dual-coupled stream block (e.g. within an affine coupling layer), wherein the (corresponding) dual-coupled stream block is configured to apply a first affine transformation (whose transformation coefficients may be determined, for example, using a neural network, e.g. using a subnetwork providing s1 (e.g. s1) and t1 (e.g. t1)) to a first part (e.g. x1, e.g. x1) of an input signal (e.g. x) to be modified by the (corresponding) dual-coupled stream block, and wherein the (corresponding) dual-coupled stream block is configured to apply a second affine transformation (whose transformation coefficients may be determined, for example, using a neural subnetwork providing s2 (e.g. s2) and t2 (e.g. t2)) to a second part (which may be different from the first part) (e.g. x2, e.g. x2) of the input signal (e.g. x) to be modified by the (corresponding) dual-coupled stream block.

[0094] According to an embodiment of the present invention, a device is provided for providing neural network parameters for audio (e.g., speech) processing (such as, for example, edge weights (θ) of a neural network that provides a scaling factor (s) and an offset value (t) based on a portion of a clean audio signal (e.g., x1, e.g., x1) or a processed version thereof, and based on a distorted audio signal (y) in a training mode, and the edge weights (θ) may correspond to the edge weights of a neural network that provides a scaling factor (s) and an offset value (t) based on a portion of a noise signal (z) or a processed version thereof and based on an input audio signal (y) in an inference mode), wherein the device is configured to (e.g., in multiple iterations) process a training audio (e.g., speech) signal (e.g., x) or a processed version thereof using one or more stream blocks (e.g., using a stream block system, e.g., including affine coupling layers, e.g., including reversible convolutions) to obtain a training result signal (which should be equal to the noise signal).

[0095] Furthermore, the apparatus is configured to adapt processing performed using one or more stream blocks according to a distorted version (e.g. y) of a training audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y) and using a neural network (which provides one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream block based on the distorted version of the training audio signal and preferably also based on at least a portion of the training audio signal or a processed version thereof).

[0096] In addition, the device is configured to determine neural network parameters of the neural network (e.g., using an evaluation of a cost function, such as an optimization function; for example, using a parameter optimization procedure) so that characteristics (e.g., probability distribution) of the training result audio signal approximate or include predetermined characteristics (e.g., noise-like characteristics; such as a Gaussian distribution).

[0097] Furthermore, the apparatus is configured to obtain a preprocessed representation (e.g., a “conditioned signal representation”) of a distorted version of the training audio signal using a filter bank (e.g., a “conditioned signal representation”, wherein, for example, the distorted version of the training audio signal can be used as a conditioned signal for adapting processing performed using one or more stream blocks), the filter bank comprising a time resolution and / or frequency resolution adapted to the time resolution and / or frequency resolution of the human auditory system (e.g., using an all-pole gammatone filter bank) (which can, for example, comprise a set of bandpass filters, wherein, for example, the bandpass filters can be infinite impulse response bandpass filters) (wherein, for example, the time / resolution and / or frequency resolution of the individual filters of the filter bank can be adapted to the time resolution and / or frequency resolution of the human auditory system).

[0098] Furthermore, the neural network is configured to receive a preprocessed representation of a distorted version of the training audio signal (e.g., as an input value) (and to provide one or more processing parameters to the stream block based on the preprocessed representation of the distorted version of the training audio signal).

[0099] According to an embodiment of the present invention, the apparatus is configured to obtain a characteristic of the training result signal (for example, according to the distribution of the obtained noise signal, such as a Gaussian function distribution, and the variance δ of the obtained noise signal). 2 ) (and optionally processing parameters, such as processing parameters of a stream block, which may, for example, be based on an input signal of the corresponding stream block) to evaluate a cost function (e.g., a loss function). Furthermore, the apparatus is configured to determine neural network parameters to reduce or minimize a cost defined by the cost function.

[0100] According to an embodiment of the present invention, a training audio signal (e.g., x) and / or a distorted version of a training audio signal (e.g., y) is represented by a set of time-domain audio samples (e.g., noisy time-domain audio (e.g., speech) samples, such as time-domain speech utterances) (wherein, for example, the time-domain audio samples of an input audio signal or time-domain audio samples derived therefrom are optionally input into a neural network) (wherein, for example, the time-domain audio samples of an input audio signal or time-domain audio samples derived therefrom are optionally input into a neural network) (wherein, for example, the time-domain audio samples of the training audio signal or time-domain audio samples derived therefrom are optionally input into a neural network) (wherein, for example, the time-domain audio samples of the training audio signal or time-domain audio samples derived therefrom are processed in the neural network in the form of a time-domain representation without applying a transform to a transform domain representation (e.g., a spectral domain representation)).

[0101] According to an embodiment of the present invention, a neural network associated with a given stream block (a given stage of affine processing) of one or more stream blocks is configured to determine one or more processing parameters (e.g. s, t) of the given stream block based on a training audio signal (e.g. x) or a signal derived from the training audio signal and based on a distorted version of the training audio signal (e.g. y).

[0102] According to an embodiment of the present invention, a neural network associated with a given stream block (e.g., a given stage of affine processing) is configured to provide one or more parameters (e.g., s, t) of an affine processing (e.g., in an affine coupling layer) that is applied to a training audio signal (e.g., x) during processing, or to a processed version of a training audio signal during processing, or to a portion of a training audio signal during processing, or to a portion of a processed version of a training audio signal during processing.

[0103] According to an embodiment of the present invention, a neural network associated with a given stream block (a given stage of affine processing) is configured to determine one or more parameters (s, t) of the affine processing based on a first part (x1, e.g. x1) of the stream block input signal (x) or based on a first part of the preprocessed stream block input signal (e.g. x') and based on a distorted version (e.g. y) of the training audio signal. In addition, the affine processing associated with the given stream block (e.g. a given stage of affine processing) is configured to apply the determined parameters to a second part (x2, e.g. x2) of the stream block input signal (x) or to a second part of the preprocessed stream block input signal (x') to obtain an affine processed signal (x2^, e.g. ). In addition, the first part (x1, such as x1) of the stream block input signal (x) or the first part of the preprocessed stream block input signal (x') (not modified by the affine processing) and the affine processed signal (x2^, such as ) forms (e.g., constitutes) the stream block output signal (x) of a given stream block (a given stage of affine processing) new ) (e.g. phase output signal).

[0104] According to an embodiment of the present invention, the device is configured to apply a reversible convolution (e.g., a 1x1 reversible convolution) to a stream block input signal (x) (stage input signal) (which can, for example, be a training audio signal of a first stage or a signal derived from a training audio signal, and which can, for example, be an output signal of a previous stage (for other subsequent stages after the first stage)) of a given stream block (e.g., a given stage of affine processing) to obtain a preprocessed stream block input signal (x') (a preprocessed version of the stream block input signal, e.g., a convolved version of the stream block input signal).

[0105] According to an embodiment of the invention, the apparatus is configured to apply a non-linear input companding (eg a non-linear compression, such as a μ-law transform) to the training audio signal (x) before processing the training audio signal (x).

[0106] According to an embodiment of the invention, the apparatus is configured to apply a μ-law transform (eg a μ-law function) as a non-linear input compression and expansion to the training audio signal (x).

[0107] According to an embodiment of the present invention, the device is configured to The transform is applied to the training audio signal (x), where sgn() is the sign function and μ is a parameter defining the compression level.

[0108] According to an embodiment of the invention, one or more stream blocks are configured to convert a training audio signal into a training result signal (which optionally approximates a noise signal, or which includes noise-like features).

[0109] According to an embodiment of the present invention, using affine processing of sample values ​​of a training audio signal or affine processing of sample values ​​of a signal derived from the training audio signal, one or more stream blocks are adjusted (e.g., by appropriate determination of neural network parameters) to convert the training audio signal into a training result signal under the guidance of a distorted version of the training audio signal (e.g., speech) signal. In addition, the processing parameters (e.g., s, t) of the affine processing are determined based on the (time domain) sample values ​​of the distorted version of the training audio signal using a neural network.

[0110] According to an embodiment of the invention, the apparatus is configured to perform normalized stream processing to derive a training result signal from a training audio signal (eg under the guidance of a distorted version of the training audio signal).

[0111] An embodiment according to the present invention includes a method for providing a processed audio (e.g., voice) signal (e.g., an enhanced audio signal) (e.g., an enhanced speech signal or an enhanced general audio signal; e.g., x^) based on an input audio (e.g., speech) signal (e.g., a distorted audio signal, e.g., a noisy speech signal y, e.g., a clean signal x extracted from the noisy speech signal y, y=x+n (noise, e.g., a noisy background)). The method includes using one or more stream blocks (e.g., using a stream block system, e.g., including an affine coupling layer, e.g., including a reversible convolution) to process (e.g., using affine scaling or using a series of affine scaling operations) a noise signal (e.g., z) or a signal derived from the noise signal to obtain a processed (e.g., enhanced) audio signal (e.g., x^). Furthermore, the method comprises adapting the processing performed using one or more stream blocks according to the input audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y; e.g. according to a noisy time-domain speech sample) and using a neural network (which provides one or more processing parameters, e.g. parameters of an affine process, such as a scaling factor and an offset value, to the stream block based on the distorted audio signal and preferably also based on at least a portion of the noise signal or a processed version thereof). Furthermore, the method comprises applying a depthwise separable convolution to a representation of the input audio signal to obtain a preprocessed representation of the input audio signal (wherein the preprocessed version of the input audio signal may, for example, comprise (per input audio signal sample) a plurality of convolution values, where the convolution values ​​are, for example, the result of different convolutions of the representation of the input audio signal with different convolution kernels). Furthermore, the neural network receives the preprocessed representation of the input audio signal (and provides one or more processing parameters to the stream block based on the preprocessed representation of the input audio signal).

[0112] According to an embodiment of the present invention, a method is provided for providing a processed audio (e.g., speech) signal (i.e., an enhanced audio signal) (e.g., an enhanced speech signal or an enhanced general audio signal; e.g., x^) based on an input audio (e.g., speech) signal (e.g., a distorted audio signal, e.g., a noisy speech signal y, e.g., a clean signal x extracted from the noisy speech signal y, y=x+n (noise, e.g., a noisy background)). The method comprises processing (e.g. using affine scaling or using a series of affine scaling operations) a noise signal (e.g. z) or a signal derived from the noise signal using one or more stream blocks (e.g. using a stream block system, e.g. including an affine coupling layer, e.g. including a reversible convolution) to obtain a processed (e.g. enhanced) audio signal (e.g. x^), and adapting the processing performed using the one or more stream blocks based on the input audio signal (e.g. a distorted audio signal, e.g. based on a noisy speech signal y; e.g. based on a noisy time domain speech sample) and using a neural network (which provides the stream block with one or more processing parameters, e.g. parameters of the affine processing, such as scaling factors and offset values, based on the distorted audio signal and preferably also based on at least a portion of the noise signal or a processed version thereof).

[0113] In addition, one or more flow blocks include at least one dual-coupled flow block (e.g., within an affine coupling layer); wherein the (corresponding) dual-coupled flow block applies a first affine transformation (whose transformation coefficients can, for example, be determined using a neural network, for example, using a subnetwork providing s1 and t1) to a first part (e.g., x1) of the input signal (e.g., x) to be modified by the (corresponding) dual-coupled flow block, and wherein the (corresponding) dual-coupled flow block applies a second affine transformation (whose transformation coefficients can, for example, be determined using a neural network, for example, using a neural subnetwork providing s2 and t2) to a second part (which can be different from the first part) (e.g., x2) of the input signal (e.g., x) to be modified by the (corresponding) dual-coupled flow block.

[0114] An embodiment according to the present invention includes a method for providing a processed audio (e.g., speech) signal (e.g., an enhanced audio signal) (e.g., an enhanced speech signal or an enhanced general audio signal; e.g., x^) based on an input audio (e.g., speech) signal (e.g., a distorted audio signal, e.g., a noisy speech signal y, e.g., a clean signal x extracted from the noisy speech signal y, y=x+n (noise, e.g., a noisy background)), wherein the method includes using one or more stream blocks (e.g., using a stream block system, e.g., including an affine coupling layer, e.g., including a reversible convolution) to process (e.g., using affine scaling, or using a series of affine scaling operations) a noise signal (e.g., z) or a signal derived from the noise signal to obtain a processed (e.g., enhanced) audio signal (e.g., x^). The method comprises adapting the processing performed using one or more stream blocks according to an input audio signal (e.g. a distorted audio signal, e.g. a noisy speech signal y; e.g. a noisy time-domain speech sample) and using a neural network (which provides one or more processing parameters, e.g. parameters of an affine process, such as a scaling factor and an offset value, to the stream block based on the distorted audio signal and preferably also based on at least a portion of the noise signal or a processed version thereof. In addition, the method comprises obtaining a preprocessed representation (e.g. a "conditioned signal representation", wherein, for example, the input audio signal can be used as a conditioned signal for adapting the processing performed using one or more stream blocks) of the input audio signal using a filter bank (e.g. using an all-pole gammatone filter bank), wherein the filter bank comprises a time resolution and / or a frequency resolution adapted to the time resolution and / or the frequency resolution of the human auditory system (the filter bank may, for example, comprise a set of bandpass filters, wherein, for example, the bandpass filters may be infinite impulse response bandpass filters) (wherein, for example, the time / resolution and / or the frequency resolution of the individual filters of the filter bank may be adapted to the time resolution and / or the frequency resolution of the human auditory system). Furthermore, the neural network receives a pre-processed representation of the input audio signal (eg, as an input value) (and provides one or more processing parameters to the stream block based on the pre-processed representation of the input audio signal).

[0115] Embodiments include a method for providing neural network parameters for audio (e.g., speech) processing (such as, for example, edge weights (θ) of a neural network that provides scaling factors (s) and offset values ​​(t) based on a portion of a clean audio signal (e.g., x1) or a processed version thereof, and based on a distorted audio signal (y) in a training mode, the edge weights (θ) may correspond to edge weights of a neural network that provides scaling factors (s) and offset values ​​(t) based on a portion of a noise signal (z) or a processed version thereof, and based on an input audio signal (y) in an inference mode), wherein the method includes processing (e.g., in multiple iterations) a training audio (e.g., speech) signal (e.g., x) or a processed version thereof using one or more streaming blocks (e.g., using a streaming block system, e.g., including affine coupling layers, e.g., including reversible convolutions) to obtain a training result signal (which should be equal to the noise signal). Furthermore, the method comprises adapting the processing performed using one or more stream blocks according to a distorted version (e.g. y) of the training audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y) and using a neural network (which is based on the distorted version of the training audio signal and preferably also based on at least a portion of the training audio signal or a processed version thereof, providing one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream blocks).

[0116] Furthermore, the method includes determining neural network parameters of the neural network (e.g., using an evaluation of a cost function, such as an optimization function; such as using a parameter optimization procedure) such that a characteristic (e.g., a probability distribution) of the training result audio signal approximates or includes a predetermined characteristic (e.g., a noise-like characteristic; such as a Gaussian distribution)

[0117] Furthermore, the method comprises applying a depthwise separable convolution to a representation of a distorted version of the training audio signal (which may, for example, replace the input audio signal during training) to derive a preprocessed representation of the distorted version of the training audio signal (wherein the preprocessed distorted version of the training audio signal may, for example, comprise (for each sample of the distorted version of the training audio signal) a plurality of convolution values, where the convolution values ​​are, for example, results of different convolutions of the representation of the distorted training audio signal with different convolution kernels).

[0118] In addition, the neural network receives a preprocessed representation of the distorted version of the training audio signal (and provides one or more processing parameters to the stream block based on the preprocessed representation of the distorted version of the training audio signal).

[0119] Embodiments include a method for providing neural network parameters for audio (e.g., speech) processing (such as, for example, edge weights (θ) of a neural network that provides scaling factors (s) and offset values ​​(t) based on a portion of a clean audio signal (e.g., x1) or a processed version thereof and based on a distorted audio signal (y) in a training mode, the edge weights (θ) may correspond to edge weights of a neural network that provides scaling factors (s) and offset values ​​(t) based on a portion of a noise signal (z) or a processed version thereof and based on an input audio signal (y) in an inference mode), wherein the method includes processing a training audio (e.g., speech) signal (e.g., x) or a processed version thereof using one or more streaming blocks (e.g., using a streaming block system, e.g., including affine coupling layers, e.g., including reversible convolutions) (e.g., in multiple iterations) to obtain a training result signal (which should be equal to the noise signal).

[0120] Furthermore, the method comprises adapting the processing performed using one or more stream blocks according to a distorted version (e.g. y) of the training audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y) and using a neural network (which provides one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream blocks based on the distorted version of the training audio signal and preferably also based on at least a portion of the training audio signal or a processed version thereof).

[0121] Furthermore, the method includes determining neural network parameters of the neural network (e.g., using an evaluation of a cost function, such as an optimization function; e.g., using a parameter optimization procedure) such that characteristics (e.g., a probability distribution) of a training result audio signal approximate or include predetermined characteristics (e.g., noise-like characteristics; e.g., a Gaussian distribution).

[0122] In addition, one or more flow blocks include at least one dual-coupled flow block (e.g., within an affine coupling layer); wherein the (corresponding) dual-coupled flow block applies a first affine transformation (whose transformation coefficients can, for example, be determined using a neural network, for example, using a subnetwork providing s1 and t1) to a first part (e.g., x1) of the input signal (e.g., x) to be modified by the (corresponding) dual-coupled flow block, and wherein the (corresponding) dual-coupled flow block applies a second affine transformation (whose transformation coefficients can, for example, be determined using a neural network, for example, using a neural subnetwork providing s2 and t2) to a second part (which can be different from the first part) (e.g., x2) of the input signal (e.g., x) to be modified by the (corresponding) dual-coupled flow block.

[0123] Embodiments include a method for providing neural network parameters for audio (e.g., speech) processing (such as, for example, edge weights (θ) of a neural network that provides scaling factors (s) and offset values ​​(t) based on a portion of a clean audio signal (e.g., x1) or a processed version thereof and based on a distorted audio signal (y) in a training mode, the edge weights (θ) may correspond to edge weights of a neural network that provides scaling factors (s) and offset values ​​(t) based on a portion of a noise signal (z) or a processed version thereof and based on an input audio signal (y) in an inference mode), wherein the method includes processing a training audio (e.g., speech) signal (e.g., x) or a processed version thereof using one or more streaming blocks (e.g., using a streaming block system, e.g., including affine coupling layers, e.g., including reversible convolutions) (e.g., in multiple iterations) to obtain a training result signal (which should be equal to the noise signal).

[0124] Furthermore, the method comprises adapting the processing performed using one or more stream blocks according to a distorted version (e.g. y) of the training audio signal (e.g. a distorted audio signal, e.g. according to a noisy speech signal y) and using a neural network (which is optionally based on the distorted version of the training audio signal and preferably also based on at least a portion of the training audio signal or a processed version thereof, providing one or more processing parameters, e.g. parameters of an affine processing, such as scaling factors and offset values, to the stream blocks).

[0125] Furthermore, the method includes determining neural network parameters of the neural network (e.g., using an evaluation of a cost function, such as an optimization function; e.g., using a parameter optimization procedure) such that characteristics (e.g., a probability distribution) of a training result audio signal approximate or include predetermined characteristics (e.g., noise-like characteristics; e.g., a Gaussian distribution).

[0126] Furthermore, the method comprises obtaining a preprocessed representation (e.g. a “conditioned signal representation”, wherein, for example, the distorted version of the training audio signal can be used as a conditioned signal for adapting processing performed using one or more stream blocks) of a distorted version of the training audio signal using a filter bank (e.g. using an all-pole gammatone filter bank), the filter bank comprising a time resolution and / or frequency resolution adapted to the time resolution and / or frequency resolution of the human auditory system (the filter bank may, for example, comprise a set of bandpass filters, wherein, for example, the bandpass filters may be infinite impulse response bandpass filters) (wherein, for example, the time / resolution and / or frequency resolution of the individual filters of the filter bank may be adapted to the time resolution and / or frequency resolution of the human auditory system).

[0127] Furthermore, the neural network receives (eg, as an input value) a preprocessed representation of a distorted version of the training audio signal (and is configured to provide one or more processing parameters to the stream block based on the preprocessed representation of the distorted version of the training audio signal).

[0128] A further embodiment comprises a computer program having a program code for performing any of the methods disclosed herein when the program is run on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0129] The accompanying drawings are not necessarily drawn to scale, with emphasis generally being placed on illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which:

[0130] Figure 1 A block diagram of a method for providing electrical connection according to an embodiment of the present invention is shown;

[0131] Figure 2 A schematic diagram showing a device with an optional all-pole gammatone filter bank according to an embodiment of the first aspect of the present invention is shown;

[0132] Figure 3 A schematic diagram showing an apparatus for providing a processed audio signal according to an embodiment of a second aspect of the present invention;

[0133] Figure 4 A comparison is shown between the conventional convolution according to an embodiment of the second aspect of the present invention and the separable convolution in the depth direction;

[0134] Figure 5 A schematic diagram of a device with an optional dual-coupled flow block according to an embodiment of the second aspect of the present invention is shown;

[0135] Figure 6 A schematic diagram showing an apparatus for providing a processed audio signal according to an embodiment of a third aspect of the present invention;

[0136] Figure 7 A schematic diagram showing an apparatus according to an embodiment of a third aspect of the present invention having additional optional features;

[0137] Figure 8 A schematic diagram of an apparatus for providing neural network parameters for audio processing according to an embodiment of the present invention is shown;

[0138] Fig. 9 A schematic block diagram showing a method for providing a processed audio signal based on an input audio signal according to an embodiment of a second aspect of the present invention;

[0139] Fig.10A schematic block diagram showing a method for providing a processed audio signal based on an input audio signal according to an embodiment of a third aspect of the present invention;

[0140] Fig.11 A schematic block diagram showing a method for providing a processed audio signal based on an input audio signal according to an embodiment of a first aspect of the present invention;

[0141] Fig.12 A schematic block diagram of a method for providing neural network parameters for audio processing according to an embodiment of a second aspect of the present invention is shown;

[0142] Fig.13 A schematic block diagram of a method for providing neural network parameters for audio processing according to an embodiment of a third aspect of the present invention is shown;

[0143] Fig.14 A schematic block diagram of a method for providing neural network parameters for audio processing according to an embodiment of the first aspect of the present invention is shown;

[0144] Fig.15 A schematic diagram of a dual coupling scheme according to an embodiment of the present invention is shown;

[0145] Fig.16 A schematic diagram showing an example of the magnitude of a filter response of an all-pole gammatone filter group (APG) according to an embodiment of the present invention;

[0146] Fig.17a A table showing an example of experimental results obtained from the VoiceBank-DEMAND test set according to an embodiment of the present invention;

[0147] Fig.17b A schematic diagram showing the hearing test results;

[0148] Fig.18 shows a schematic visualization of the basic principles of a training process according to an embodiment of the present invention;

[0149] Fig.19 A schematic visualization of the basic principles of the training process and the enhancement process according to an embodiment of the present invention is shown;

[0150] Fig. 20 shows a schematic visualization of improvements (eg, a first improvement) according to an embodiment of the present invention;

[0151] Fig.21 shows a schematic visualization of a single coupling scheme according to an embodiment of the present invention;

[0152] Fig. 22 shows a schematic visualization of a dual coupling scheme according to an embodiment of the present invention;

[0153] Fig.23 A schematic diagram showing the improved principle of the training process and the enhancement process according to an embodiment of the present invention; and

[0154] Fig.24 Another schematic diagram showing hearing test results. DETAILED DESCRIPTION

[0155] In the following description, the same or equivalent elements or elements having the same or equivalent functions are denoted by the same or equivalent reference numerals even if they appear in different drawings.

[0156] In the following description, a number of details are provided to provide a more comprehensive explanation of embodiments of the present invention. However, it will be appreciated by those skilled in the art that embodiments of the present invention may be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present invention. In addition, unless otherwise specifically stated, the features of the different embodiments described below may be combined with each other.

[0157] Figure 1 A schematic diagram of an apparatus for providing a processed audio signal according to an embodiment of the first aspect of the present invention is shown. Figure 1 An apparatus 100 is shown, which is provided with an input audio signal 101 (e.g. y) and a noise signal 102 (e.g. z). It is noted that the noise signal 102 may be, for example, a signal derived from the noise signal. The apparatus 100 is configured to provide a processed audio signal 103, e.g. It may optionally be an enhanced signal, such as an enhanced audio signal, eg an enhanced version of the input audio signal 101 .

[0158] The apparatus 100 comprises a stream block 110, a neural network 120 and a filter bank 130. The stream block 110 is provided with a noise signal 102. Using the stream block 110, the apparatus 100 is configured to provide a processed audio signal 103 based on processing the noise signal 102 (or a corresponding signal derived from the noise signal). Therefore, the stream block 110 may optionally include an affine coupling layer, and / or a reversible convolution.

[0159] Furthermore, the device 100 is configured to adapt the processing performed using the stream block 110 based on the input audio signal 101 and using the neural network 120 (and optionally based on the noise signal or a portion thereof). As shown, optionally, the neural network 120 can be configured to provide information 121 for adapting the processing performed using the stream block 110. Thus, the apparatus 100 can adapt the processing via the neural network based on the input audio signal 120. Optionally, as shown, the input audio signal 101 can be provided directly to the stream block 110 to adapt the processing, for example to adapt to an affine transformation performed in the stream block 110.

[0160] Thus, the neural network 120 is configured to receive the preprocessed representation 131 of the input audio signal. As an example, based on the preprocessed representation 131, the neural network may provide to the stream 110 via the information 121 one or more parameters for processing the noise signal 102 in the stream block 110.

[0161] Furthermore, the apparatus 100 is configured to obtain a preprocessed representation 131 of the input audio signal using a filter bank 130. The filter bank 130 comprises a time resolution and / or a frequency resolution adapted to the time resolution and / or the frequency resolution of the human auditory system.

[0162] Simplified, the apparatus 100 may be provided with an input signal 101, e.g. y=x+n, which is distorted and / or noisy, and thus comprises a clean speech portion x and its distortion, and / or noise portion, e.g. n. Furthermore, the apparatus 100 may be provided with a noise signal 102, e.g. z, e.g. sampled from a Gaussian distribution, e.g. with zero mean and unit variance. The apparatus 100 may use a stream block 110 (or use a stream block chain) to shape the noise signal z to provide an enhanced version of the input signal 101 as a processed audio signal, e.g. Thus, the stream block 110 may, for example, include an affine coupling layer, and / or may be configured to perform an affine transformation, for example using a reversible convolution. In order to control the processing of the noise signal 102 using the stream block 110, the input signal 101 may be processed by the neural network 120 to provide parameters for performing the processing, for example in the form of an affine transformation. Thus, the neural network 120 may be used to provide parameters for such a transformation, for example using adaptation information 121. The inventors have recognized that the efficiency and robustness of such processing may be improved by implementing a filter bank 130 having a temporal resolution and / or frequency resolution adapted to the temporal resolution and / or frequency resolution of the human auditory system as a pre-processing unit of the neural network. Thus, improved parameters may be provided for the stream block processing, thereby improving the quality of the processed (e.g. enhanced) signal 103.

[0163] like Figure 1As shown, the neural network 120 may optionally be provided with the noisy signal 102 (or a signal derived therefrom) to provide adaptation information 121 .

[0164] As an optional feature, the time resolution of the filter bank 130 and the frequency resolution of the filter bank 130 approximate the time resolution and frequency resolution of the human auditory system. As an example, the filter bank 130 can be designed to mimic human hearing characteristics or designed to represent human hearing characteristics and / or designed to take human hearing characteristics into account to improve the pre-processing of the input audio signal 101 of the neural network 120.

[0165] As another optional feature, the filter bank 130 may include one or more filters, and the filters or some of the filters may include or may be infinite impulse response filters.

[0166] Figure 2 A schematic diagram of a device with a selectable all-pole gammatone filter group according to an embodiment of the first aspect of the present invention is shown. Figure 2 An apparatus 200 is shown, which includes a stream block 210, a neural network 220 and a filter bank 230, having at least the following non-optional features and functions: Figure 1 However, it should be noted that the apparatus 200 may include the following: Figure 1 Optional features and functionality are shown in the context of the .

[0167] The apparatus 200 is provided with a noise signal 202 (which is optionally a signal derived from the noise signal) and with an input audio signal 201, and provides a processed audio signal 203, such as Figure 1 as shown in the context.

[0168] As an optional feature, the stream block 210 comprises a neural network 220. Furthermore, as an optional feature, an optional affine transformation unit 240 as part of the stream block 210 is used to provide a processed audio signal 203. As shown, the neural network optionally provides adaptation information 221 to the affine transformation unit 240, such as parameters for processing of the noise signal 202. The affine transformation unit 240 may optionally comprise an affine coupling layer and / or a reversible convolution for providing a processed audio signal 203 based on the noise signal 202 and optionally based on the input audio signal 201.

[0169] like Figure 2 As shown, the neural network 220 may optionally be provided with the noisy signal 202 (or a signal derived therefrom) to provide adaptation information 221 .

[0170] Furthermore, as another optional feature, the apparatus 200 includes a pre-processing unit 250 between the filter bank 230 and the neural network 220 .

[0171] As another optional feature, the filter bank 230 is an all-pole gamma filter bank, such as a complex all-pole gamma filter bank. Alternatively, the filter bank 230 may be a specially designed complex-valued all-pole gamma filter bank, such as an APG, such as an APGFB.

[0172] Note that filter bank 230 , being an all-pole gammatone filter bank, may be used with or without the optional pre-processing unit, and may also be used in configurations where neural network 220 is not part of stream block 210 .

[0173] Optionally, the design of the filter bank 230 can be inspired by human hearing, so that the center frequencies of the optional IIR filters of the filter bank 230 can optionally have a constant distance on the Bark scale, for example, as the frequency increases, the bandwidth also increases, which can be, for example, proportional to the Bark bandwidth. In order to compensate for possible time differences in the filter outputs, the look-ahead of each filter can be implemented, for example, according to the group delay of each filter at the center frequency, for example, scaled with a common factor for all frequency bands. For example, only the amplitudes produced by the filter bank 230 can optionally be further processed by the network 220.

[0174] As an optional feature, the all-pole gammatone filter bank 230 is configured to obtain a plurality of channel signals associated with a plurality of frequency bands, wherein the width of the frequency bands increases monotonically with increasing center frequencies of the corresponding frequency bands, and / or wherein the width of the frequency bands is adapted according to a psychoacoustic model and / or the center frequencies of the frequency bands are adapted according to a psychoacoustic model. As an optional feature, the input audio signal 201 may include said channel signals associated with a plurality of frequency bands.

[0175] Additionally, as an optional feature, all-pole gammatone filter bank 230 includes a plurality of filters wherein the center frequencies of the filters comprise a constant distance on the Bark scale and the bandwidth increases as the frequency increases.

[0176] Furthermore, as an optional feature, all-pole gammatone filter bank 230 is configured to at least partially compensate for different group delays between different filters.

[0177] Optionally, the transfer function of the all-pole gamma filter bank does not include any finite zeros. In addition, as another optional feature, the imaginary parts of the poles of the transfer function of the all-pole gamma filter bank 230 all include the same sign. In addition, optionally, the all-pole gamma filter bank 230 is a complex all-pole gamma filter bank. As another optional feature, one or more poles of the transfer function of the all-pole gamma filter bank 230 overlap.

[0178] As another optional feature, the apparatus 100 is configured to obtain the preprocessed representation 131 of the input audio signal based on the amplitude of the output signal of the all-pole gammatone filter bank. In this case, the apparatus 100 is configured to ignore the phase information of the output signal of the all-pole gammatone filter bank.

[0179] As another optional feature, the all-pole gammatone filter bank is configured to provide 20 to 100 output signals.

[0180] As another optional feature, the device 200 is configured to apply multiple convolutions to a set of output values ​​of the all-pole gammatone filter bank 230 or to a set of amplitude values ​​derived from the output values ​​of the all-pole gammatone filter bank 230, for example using a preprocessing unit 250, to obtain input values ​​251 of the neural network.

[0181] like Figure 2 As shown, the output signal 231 of the all-pole gammatone filter bank 230 is provided to an optional pre-processing unit 250, which is configured to apply a convolution to the output and / or amplitude values. Based on this, input values ​​251 of the neural network 220 are provided.

[0182] As another optional feature, the apparatus 200 is configured to apply a depthwise separable convolution to a set of output values ​​of the all-pole gammatone filter bank 230 or to a set of amplitude values ​​derived from the output values ​​of the all-pole gammatone filter bank 230 to obtain input values ​​of the neural network. Therefore, the processing in the preprocessing unit 250, as an optional feature, includes the application of a depthwise separable convolution.

[0183] Therefore, if Figure 1 As shown, the pre-processed representation of the input audio signal 131 may comprise the input value 251 , or may even be said input value.

[0184] Figure 3 A schematic diagram of an apparatus for providing a processed audio signal according to an embodiment of a second aspect of the present invention is shown. Figure 3An apparatus 300 is shown, which is provided with an input audio signal 301 (e.g. y) and a noise signal 302 (e.g. z). It is noted that the noise signal 302 may be, for example, a signal derived from the noise signal. The apparatus 300 is configured to provide a processed audio signal 303, e.g. It may optionally be an enhanced signal, such as an enhanced audio signal, such as an enhanced version of the input audio signal 301 .

[0185] The apparatus 300 comprises a stream block 310, a neural network 320 and a pre-processing unit 330. The stream block 310 is provided with a noise signal 302. Using the stream block 310, the apparatus 300 is configured to provide a processed audio signal 303 based on processing of the noise signal 302 (or a corresponding signal derived from the noise signal).

[0186] Furthermore, the apparatus 300 is configured to adapt the processing performed using the stream block 310 according to the input audio signal 301 and using the neural network 320. As shown, optionally, the neural network 320 may be configured to provide information 321 for adapting the processing performed using the stream block 310. Thus, the apparatus 300 may adapt the processing via the neural network 320 according to the input audio signal, and / or, as an alternative, the input audio signal 301 shown may be provided directly to the stream block 310 to adapt the processing, for example, to adapt to an affine transformation performed in the stream block 310.

[0187] Thus, the neural network 320 is configured to receive the preprocessed representation 331 of the input audio signal. As an example, based on the preprocessed representation 331, the neural network may provide one or more parameters for processing the noise signal 302 in the stream block 310 to the stream block 310 via the information 321.

[0188] Furthermore, the apparatus 300 is configured to apply a depthwise separable convolution to the representation of the input audio signal to derive a preprocessed representation of the input audio signal 331. The application of the convolution is performed in a preprocessing unit.

[0189] Using depthwise separable convolutions, a reduction in parameters (e.g., for performing affine transformations) and / or in complexity may be achieved.

[0190] like Figure 3 As shown, the neural network 320 may optionally be provided with the noisy signal 302 (or a signal derived therefrom) to provide adaptation information 321 .

[0191] As an example, refer to Figure 4 . Figure 4 A comparison between the application of conventional convolution and depth-wise separable convolution according to an embodiment of the second aspect of the present invention is shown.

[0192] As an example, an input signal (e.g., signal 101, 201, and / or 301) may include 80 frequency bands and 8000 time samples (401). Based on this, it may be necessary to obtain 2000 output dimensions, or more precisely, in other words, "2000 different ways of transforming the input".

[0193] As an example, assume that the kernel size in time samples is 3 (403). Thus, applying a regular or normal 1d convolution to 80 bands with 8000 time samples may produce 2000 kernels of size 80x3, and thus 2000x80x3=480000 parameters (404). As optionally shown, the output dimension may be [2000x8000], e.g. assuming "same padding" or "time padding", e.g., such that the input time dimension is equal to the output time dimension. (405).

[0194] In contrast, an example of a depth-wise separable convolution (406) is shown, including depth-wise (407) and point-wise (408) convolutions. According to the depth-wise convolution (407) (e.g., as a first step or first convolution), as an example, 80 kernels with a width of 3 but a depth of 1 can be made or provided. Thus, as shown, the number of parameters can be 80x3x1=240, with an output size of [80, 8000]. According to the point-wise convolution (e.g., as a second step or second convolution), in short, the convolution can include a depth of 80 but only a width of 1.

[0195] As defined previously (402), the output dimension should be 2000. Therefore, the number of parameters is 80x1x2000=160000, resulting in an output dimension of [2000x8000].

[0196] Adding the number of parameters for the convolutions in the depth direction (407) and the point direction (408) yields a total of 160,240 parameters, compared to 480,000 parameters according to the conventional method (409).

[0197] In other words, based on the depthwise separable convolution, based on the same input signal and producing the same output dimension, the number of parameters required for the corresponding convolution or continuous convolution can be significantly reduced.

[0198] Figure 5 A schematic diagram of a device with an optional dual-coupled flow block according to an embodiment of the second aspect of the present invention is shown. Figure 5 The apparatus 500 is shown, which includes a flow block 510, a neural network 520 and a pre-processing unit 530, having at least the non-optional features and functions as shown in FIG. Figure 3However, it should be noted that the apparatus 500 may include the following: Figure 1 , 2 , 3 and 4 are optional features and functions shown in the context of.

[0199] The apparatus 500 is provided with a noise signal 502 (which is optionally a signal derived from the noise signal) and with an input audio signal 501, and provides a processed audio signal 503. Figure 5 As shown, neural network 520 may optionally be provided with noisy signal 502 (or a signal derived therefrom) to provide adaptation information 521 .

[0200] As such an optional feature, the apparatus 500 comprises a filter bank 540. As an optional feature, the apparatus 500 is configured to obtain a preprocessed representation of the input audio signal 501 using the filter bank 540, the filter bank 540 comprising a temporal resolution and / or a frequency resolution adapted to human hearing. As shown, optionally, the neural network is configured to receive the preprocessed representation of the input audio signal. The neural network 520 may be configured to be directly provided with the preprocessed representation of the filter bank 540, however, as shown Figure 5 As shown in the example of , the output of the filter bank 540 can be further processed, for example by a preprocessing unit 531, so as to provide a preprocessed representation 531 of the input audio signal to the neural network 520.

[0201] Furthermore, as another optional feature, the apparatus 500 is configured to obtain a preprocessed representation of the input audio signal using an all-pole gammatone filter bank, and the neural network 520 is configured to receive the preprocessed representation of the input audio signal (as shown in the optional feature) via the preprocessing unit 530. In other words, Figure 5 In the example of , the filter bank 540 is an all-pole gammatone filter bank. Optionally, the all-pole gammatone filter bank may include Figure 1 to Figure 2 Any or all features disclosed in the context of.

[0202] Furthermore, as another optional feature, the neural network 520 is provided with the noise signal 502, or optionally provided with information about the noise signal. Figures 1 to 3 In any embodiment disclosed in the context of , the provision of a corresponding noise signal may be implemented as an optional feature.

[0203] As another optional feature, as previously mentioned, the stream block 510 is a dual coupled stream block configured to apply a first affine transformation 512 to a first portion 502a of the input signal to be modified by the dual coupled stream block, e.g. z1, and to apply a second affine transformation 514 to a second portion 502b of the input signal to be modified by the dual coupled stream block, e.g. z2. In the example shown, the input signal to be modified is a noise signal 502. As shown, the noise signal 502 may optionally be split into a first and a second portion before providing the respective portions to the respective transformations. As shown, the results of the transformations may optionally be merged or, for example, concatenated to provide a processed audio signal 503, e.g.

[0204] Furthermore, as optionally shown, the results of the corresponding affine transformations 512, 521 may optionally be provided to the neural network 520. Thus, the transformed parts (e.g. For example ) can be used to determine affine transformation parameters. Specifically, the output of the first affine transformation 512 (e.g. ) can be used to determine the parameters or parameter adaptation of the second affine transformation 514, and vice versa, the output of the second affine transformation 514 (e.g. ) can be used to determine the parameters or parameter adaptation of the first affine transformation 512, for example in Figure 7 , 15 and / or as shown and discussed in the context of 22.

[0205] Thus, according to some embodiments, an important aspect may be to use the already transformed first part (e.g. from a first affine transformation) to determine (e.g. using a neural network) the affine parameters of a second affine transformation (or an adaptation thereof) (and, e.g., vice versa). This approach may ensure the reversibility of the corresponding transformations and / or the entire transformation. Thus, for example, the dual coupling scheme may be as follows Fig.15 as shown and discussed.

[0206] Furthermore, as previously described, the neural network optionally provides parameters for the transformations 512 , 514 via information 521 to adapt the transformations with respect to the corresponding input signal 501 , wherein it may be desirable to enhance the input signal in the form of signal 503 .

[0207] As an optional feature, the depthwise separable convolution is configured to perform temporal convolution and convolution in the frequency direction. As an example, the preprocessing unit 530 may be configured to apply the depthwise separable convolution, thereby performing temporal convolution and convolution in the frequency direction.

[0208] Thus, as an optional feature, the apparatus 500 is configured to obtain a representation 541 of the input audio signal using an all-pole gammatone filter bank 540, and to apply a depthwise separable convolution to the representation of the input audio signal obtained using the all-pole gammatone filter bank. This is performed via a pre-processing unit 530 (as an optional feature).

[0209] As an optional feature, the apparatus is configured to apply different convolutions to the plurality of sub-band signals to obtain the neural network's input signal 531. The representation 541 of the input audio signal obtained using the all-pole gammatone filter bank 540 may comprise the plurality of sub-band signals.

[0210] In addition, as an optional feature, the device 500 is configured to apply separate time convolutions to multiple signals representing the input audio signal to obtain multiple time convolution signal values, and is configured to apply multiple convolutions in frequency to a given set of time convolution signal values ​​to obtain multiple input values ​​531 of the neural network.

[0211] As an example, the pre-processing unit 530 may be configured to perform the following steps: Figure 4 Steps (407) to (408) are shown.

[0212] As another optional feature, the apparatus 500 is configured to apply a depthwise separable convolution to the representation (e.g. 541) of the input audio signal, e.g. using the pre-processing unit 531, to map the input space to a higher dimension. Figure 4 and Figure 5 , the pre-processing unit 531 can be configured to map the input space (eg, the dimension of the frequency band (401)) to a higher dimension, such as defined in the output dimension (405).

[0213] As another optional feature, the apparatus 500 is configured to perform multiple convolutions in frequency using the preprocessing unit 530 based on the same set of result values ​​of the respective time convolutions, wherein the respective time convolutions are respectively performed based on signals of the frequency domain representation 541 of the input audio signal 501 .

[0214] Figure 6 A schematic diagram of an apparatus for providing a processed audio signal according to an embodiment of a third aspect of the present invention is shown. Figure 6 An apparatus 600 is shown, which is provided with an input audio signal 601 (e.g. y) and a noise signal 602 (e.g. z). It is noted that the noise signal 602 may be, for example, a signal derived from the noise signal. The apparatus 600 is configured to provide a processed audio signal 603, e.g. It may optionally be an enhanced signal, such as an enhanced audio signal, such as an enhanced version of the input audio signal 601 .

[0215] The apparatus 600 comprises a stream block 610 and a neural network 620. The stream block 610 is provided with a noise signal 602. Using the stream block 610, the apparatus 600 is configured to provide a processed audio signal 603 based on processing of the noise signal 602 (or a corresponding signal derived from the noise signal).

[0216] Furthermore, the apparatus 600 is configured to adapt the processing performed using the stream block 610 according to the input audio signal 601 and using the neural network 620. As shown, optionally, the neural network 620 may be configured to provide information 621 for adapting the processing performed using the stream block 610. Thus, the apparatus 600 may adapt the processing via the neural network 620 according to the input audio signal and / or, as optionally, may provide the illustrated input audio signal 601 directly to the stream block 610 to adapt the processing, for example, to adapt to an affine transformation performed in the stream block 610.

[0217] The stream block 610 is a dual coupled stream block configured to apply a first affine transformation 612 to a first portion 602a of an input signal to be modified by the dual coupled stream block, e.g. z1, and to apply a second affine transformation 614 to a second portion 602b of the input signal to be modified by the dual coupled stream block, e.g. z2. In the example shown, the input signal to be modified is a noise signal 602. As shown, the noise signal 602 may optionally be split into a first and a second portion before providing the respective portions to the respective transformations. As shown, the results of the transformations may optionally be concatenated or, for example, merged to provide a processed audio signal 603, e.g.

[0218] Furthermore, as previously described, the neural network optionally provides parameters for the transformations 612 , 614 via information 621 in order to adapt the transformations with respect to the corresponding input signal 601 , wherein it may be desirable to enhance the input signal in the form of signal 603 .

[0219] like Figure 6 As shown, the neural network 620 may optionally be provided with the noisy signal 602 (or a signal derived therefrom) to provide adaptation information 621 .

[0220] In addition, as optionally related to Figure 5 As shown and discussed, the results of the corresponding affine transformations 612, 621 can optionally be provided to the neural network 620. Thus, the transformed portion (e.g. For example ) can be used to determine affine transformation parameters. Specifically, the output of the first affine transformation 612 can be used to determine the parameters or parameter adaptation of the second affine transformation 614, and vice versa, the output of the second affine transformation 614 can be used to determine the parameters or parameter adaptation of the first affine transformation 612, for example, Figure 7 , 15 and / or as shown and discussed in 22.

[0221] Figure 7 A schematic diagram of an apparatus according to an embodiment of the third aspect of the present invention with additional optional features is shown. Figure 7 An apparatus 700 is shown for providing a processed audio signal 703 based on an input audio signal 701 (e.g. y=x+n). The apparatus 700 is configured to process a noise signal 702 (e.g. z) or a signal derived from the noise signal using one or more stream blocks to obtain the processed audio signal 703, e.g.

[0222] exist Figure 7 In the example shown, the apparatus 700 optionally comprises first and second stream blocks 710, 720. Furthermore, the apparatus 700 is configured to adapt the processing performed using one or more stream blocks 710, 720 according to the input audio signal 701 and using a neural network.

[0223] according to Figure 7 In the example of , the adaptation using the input signal 701 is performed via a neural network in the form of a plurality of sub-networks 712, 714, 722, 724. However, as previously mentioned, the introduction of information extracted from the input audio signal 701 may be performed separately from the processing via a neural network in addition or alternatively.

[0224] like Figure 7 As shown, optionally, one or more flow blocks 710, 720 include at least one dual-coupled flow block. In the case of the device 700, as an example, both flow blocks 710, 720 are dual-coupled flow blocks.

[0225] Each dual coupled stream block is respectively configured to apply a first affine transformation to a first portion of an input signal to be modified by the dual coupled stream block and to apply a second affine transformation to a second portion of the input signal to be modified by the dual coupled stream block.

[0226] like Figure 7 As shown, the noise signal 702 is provided to the first stream block 710, where it is divided into a first portion 702a (e.g., z1) and a second portion 702b (e.g., z2). The first portion 702a is processed via a first affine transformation 716, and the second portion 702b is processed via a second affine transformation 718.

[0227] The respective sub-networks 712, 714 are provided with the input audio signal 701 to adapt the parameters of the respective affine transformation.

[0228] As previously described, as an optional feature, the corresponding dual-coupled stream blocks 710, 720 are configured to separate the corresponding input signals 702, 731 of the corresponding dual-coupled stream blocks to obtain a first part 702a, 731a of the input signal, and apply respective affine transformations to the first part of the input signal and the second part 702b, 731b of the input signal.

[0229] As mentioned above, as another optional feature, the apparatus 700 is configured to connect the output signals of the first affine transformation 716, 726 and the output signals of the second affine transformation 718, 728 to obtain the output signals of the corresponding dual coupled stream blocks.

[0230] As an optional feature, the device 700 is configured to use a neural network 712 to adapt processing to be performed by the first affine transformation 716 of the dual coupled stream block 710 based on the second affine transformed input signal 702b, and the device 700 is configured to use a neural network 714 to adapt processing to be performed by the second affine transformation 718 of the dual coupled stream block based on the first portion 702a of the input signal to be modified by the dual coupled stream block.

[0231] Thus, subnetwork 712 is provided with portion 702b and subnetwork 714 is provided with the result of first affine transformation 716. Optionally, subnetwork 714 may additionally or alternatively be provided with portion 702a and subnetwork 712 may additionally or alternatively be provided with the result of second affine transformation 718.

[0232] In other words, the apparatus 700 is configured to use the second portion 702b, 731b of the input signal as an input signal for the neural network 712, 722 for determining the transformation parameters 713, 723 of the first affine transformation 716, 726 and as an input signal for the second affine transformation 718, 728.

[0233] Furthermore, the apparatus 700 is configured to use the output signal of the first affine transformation 717 , 727 as an input signal of the neural network 714 , 724 for determining the transformation parameters 715 , 725 of the second affine transformation 718 , 728 .

[0234] Therefore, as an optional feature, the dual coupled stream blocks 710, 720 are configured to split the input signal 702, 731 to be modified by the dual coupled stream block into two halves, and are configured to use the second half of the input signal to be modified by the dual coupled stream block to estimate parameters of the affine transformation to be applied to the first half of the input signal to be modified by the dual coupled stream block.

[0235] As an optional feature, the stream block 720 is a corresponding dual coupled stream block. Thus, for example, the connected or merged results of the affine transformations 716, 718 of the first stream block 710 are provided to the second stream block 720 as input signal.

[0236] Therefore, as an optional feature, the apparatus 700 comprises a convolution unit 730. In other words, as an optional feature, the apparatus 700 is configured to apply processing using a sequence of a plurality of dual coupled stream blocks 710, 720, wherein the apparatus is configured to apply a reversible convolution using the convolution unit 730 to obtain an input signal 731 of a second dual coupled stream block 720 based on an output signal of a previous first dual coupled stream block.

[0237] exist Figure 7 In the example, as another optional feature, the dual coupled stream blocks 710, 720 are configured to modify only the signal of the first part 702a, 731a of the input signal to be modified by the dual coupled stream block in the first affine transformation 716, 726, and to modify only the signal of the second part 702b, 731b of the input signal to be modified by the dual coupled stream block in the second affine transformation 718, 728.

[0238] It should be noted that according to Figure 6 or Figure 7 The embodiments may optionally include any or all of the features, for example, in particular Figures 1 to 5 The filter bank and / or pre-processing unit aspects disclosed in the context of can be applied to any input audio signal.

[0239] In the following, additional optional features will be discussed with respect to the apparatuses 100, 200, 300, 500, 600 and 700. The apparatuses may include any or all of the following optional features, alone or in combination.

[0240] First, it is noted that the input audio signal (e.g., 101, 201, 301, 501, 601, 701) can optionally be represented by a set of time-domain audio samples. In addition, as previously described, a neural network (e.g., including one or more sub-networks (e.g., 120, 220, 320, 520, 620, 712, 714, 722, 724, 820)) associated with a given stream block in one or more stream blocks is optionally configured to determine one or more processing parameters of the given stream block based on a noise signal (e.g., 102, 202, 302, 502, 602, 702 (e.g., z)) or a signal derived from the noise signal and based on the input audio signal 101, 201, 301, 501, 601, 701 (e.g., y).

[0241] In general, depending on an embodiment, optionally, a neural network associated with a given stream block (e.g., comprising one or more sub-networks (e.g., 120, 220, 320, 520, 620, 712, 714, 722, 724, 820)) can be configured to provide one or more parameters of an affine processing that is applied to a noise signal (e.g., 102, 202, 302, 502, 602, 702) during processing, or to a processed version of a noise signal during processing, or to a portion of a noise signal during processing, or to a portion of a processed version of a noise signal during processing.

[0242] In this case, optionally, the neural network associated with a given stream block can be configured to determine one or more parameters of the affine processing, such as 715, such as 725, based on a first part or a first portion (e.g., 702a, 731a (e.g., z1)) of the stream block input signal (e.g., 702, 731 (e.g., z)) and based on the input audio signal (e.g., 701 (e.g., y)). In addition, in this case, the affine processing associated with the given stream block can be configured to apply the determined parameters to a second part or a second portion (e.g., 702b, 731b (e.g., z2)) of the stream block input signal (e.g., z) to obtain an affine processed signal (e.g., z2^). In addition, optionally, the first part (e.g., 717, 727 (e.g., z1)) of the stream block input signal (z) and the affine processed signal (e.g., 719, 729 (e.g., z2^)) can form a stream block output signal of the given stream block.

[0243] As another optional feature, the apparatus 100, 200, 300, 500, 600, 700 according to the embodiments may be configured to apply a reversible convolution (e.g. 730) to the flow block output signal (z) of a given flow block. new ) to obtain the processed stream block output signal (z' new ).

[0244] Note that for simplicity, Figure 1 , 2 , 3, 5, and 6 only show one flow block. However, as mentioned above, the apparatus according to the embodiment may include multiple flow blocks, such as Figure 7 As shown, with or without a dual coupling structure. Therefore, these flow blocks can be coupled via a convolution unit 730.

[0245] In addition, optionally, the apparatus according to the embodiments (e.g., 100, 200, 300, 500, 600, 700) may be configured to apply a nonlinear expansion to the processed audio signal, e.g., 103, 203, 303, 503, 603, 703, for example, using a corresponding stream block (not shown).

[0246] In particular, as an optional feature, the corresponding apparatus may be configured to apply an inverse μ-law transform as a nonlinear expansion to the processed audio signal (eg x^). As an example, according to The transform of can be performed on the processed audio signal (e.g. x^), where sgn() is the sign function and where μ is a parameter defining the level of expansion.

[0247] In general, it is noted that, according to an embodiment, as an optional feature, processing of a training audio signal or a processed version thereof may be used in one or more training stream blocks to obtain neural network parameters of a neural network (e.g., 120, 220, 320, 520, 620, 712, 714, 722, 724, 820) for processing a noise signal (e.g., 102, 202, 302, 502, 602, 702) or a signal derived from the noise signal to obtain a training result signal. The processing of the training audio signal or a processed version thereof using one or more training stream blocks may optionally be based on a distorted version of the training audio signal and adapted using a neural network. In addition, the neural network parameters of the neural network may be determined such that the features of the training result audio signal approximate or include predetermined features.

[0248] Furthermore, it is noted that any of the devices 100, 200, 300, 500, 600, 700 may be configured to provide neural network parameters of a neural network for processing a noise signal or a signal derived from a noise signal. Thus, the respective device may be configured to process a training audio signal or a processed version thereof using one or more stream blocks to obtain a training result signal, thereby adapting the processing of the training audio signal or a processed version thereof performed using one or more stream blocks based on the distorted version of the training audio signal and using a neural network. Furthermore, the respective device may be configured to determine the neural network parameters of the neural network so that the features of the training result audio signal approximate or include predetermined features.

[0249] Furthermore, it is noted that any of the apparatuses 100, 200, 300, 500, 600, 700 may include means for providing neural network parameters, wherein the means for providing neural network parameters is configured to provide neural network parameters of a neural network for processing a noise signal or a signal derived from a noise signal, wherein the means for providing neural network parameters is configured to process a training audio signal or a processed version thereof using one or more training stream blocks to obtain a training result signal, and wherein the means for providing neural network parameters is configured to adapt the processing of the training audio signal or a processed version thereof performed using one or more stream blocks based on a distorted version of the training audio signal using a neural network. Furthermore, the apparatus is configured to determine the neural network parameters of the neural network so that the characteristics of the training result audio signal approximate or include predetermined characteristics.

[0250] In addition, optionally, one or more stream blocks (e.g., 110, 210, 310, 510, 610, 710, 720) are configured to synthesize a processed audio signal (e.g., 103, 203, 303, 503, 603, 703) based on a noise signal (e.g., 102, 202, 302, 502, 602, 702) under the guidance of an input audio signal (e.g., 101, 201, 301, 501, 601, 701).

[0251] Alternatively, the synthesis may be performed using an affine processing of sample values ​​of the noise signal or of a signal derived from the noise signal, wherein processing parameters (e.g. s, t) of the affine processing are determined based on sample values ​​of the input audio signal using a neural network.

[0252] Furthermore, as an optional feature, the apparatus (eg, 100, 200, 300, 500, 600, 700) is configured to perform normalized stream processing to derive the processed audio signal from the noise signal.

[0253] Figure 8 A schematic diagram of a device for providing neural network parameters for audio processing according to an embodiment of the present invention is shown. The device 800 includes a stream block 810 and a neural network 820.

[0254] The apparatus 800 is configured to process a training audio signal 802 (eg, x (eg, clean speech)) or a processed version thereof using a stream block 810 to obtain a training result signal 811 .

[0255] The apparatus 800 is configured to adapt processing performed using the stream block 810 based on a distorted version 801 of a training audio signal (e.g., y, e.g., y=x+n) and using a neural network 820, e.g., by providing parameters for an affine transformation performed in the stream block 810 via adaptation information 821.

[0256] The apparatus is configured to determine a neural network parameter 803 of a neural network 820 so that a feature of a training result audio signal 811 approximates or includes a predetermined feature (eg, a predetermined feature of a noise signal z).

[0257] Therefore, the device 800 includes a neural network parameter provider 850.

[0258] Furthermore, according to the first, second and / or third aspect of the present invention, the apparatus 800 comprises one or more of the following features.

[0259] The apparatus 800 may be configured to obtain a preprocessed representation 831 of the distorted version 801 of the training audio signal using an optional filter bank 830 comprising a time resolution and / or a frequency resolution adapted to that of the human auditory system.

[0260] Additionally or alternatively, the apparatus 800 may be configured to apply a depthwise separable convolution to the representation 801 or 831 of the distorted version of the training audio signal to derive a preprocessed representation 841 of the distorted version of the training audio signal. Thus, the apparatus may include an optional preprocessing unit 840.

[0261] Thus, the neural network 820 is configured to receive respective pre-processed representations 831 , 841 of distorted versions of the training audio signals.

[0262] Additionally or alternatively, the stream block 810 may include at least one dual-coupled stream block, wherein the dual-coupled stream block is configured to apply a first affine transformation to a first portion of an input signal to be modified by the dual-coupled stream block, and wherein the dual-coupled stream block is configured to apply a second affine transformation to a second portion of the input signal to be modified by the dual-coupled stream block.

[0263] In general, it is noted that the structure of the inventive device for obtaining the training result signal can be similar or even identical to the inventive device for obtaining the processed signal. According to some embodiments, these devices can be identical, for example excluding different units for determining the parameters of the neural network. Thus, Figure 8 The illustrated embodiment may include, for example, aspects of a filter bank 830, a pre-processing unit 840, a stream block 810, and a neural network 820, as described in Figures 1 to 7 Any features disclosed in the context of .

[0264] In contrast, instead of a noise signal, for example z, a training audio signal 802 may be provided, which may be a clean speech signal. The signal 802 may be processed into a training result signal 811 via a stream block 810, the goal of which is to have specific characteristics, approximate the noise signal. The processing is guided via the distorted signal 801. Vice versa, in order to obtain a processed (e.g., enhanced) signal, the stream block transformation may be reversed to transform the noise signal into a processed signal under the guidance of the audio input signal (e.g., a distorted signal of clean speech), as an enhanced version of the distorted signal, e.g., approximate clean speech. Therefore, the same neural network may be used, e.g., a neural network having the same parameters and topology.

[0265] As an optional feature, the apparatus 800 is configured to evaluate a cost function based on the features of the obtained training result signal 811, for example using the NN parameter provider 850, and is configured to determine the neural network parameters 803 to reduce or minimize the cost defined by the cost function. This can allow the neural network parameters to be efficiently determined.

[0266] Furthermore, optionally, the training audio signal 802 and / or the distorted version 801 of the training audio signal may be represented by a set of time-domain audio samples.

[0267] Additionally, as another optional feature, a neural network (e.g., Figure 8 The neural network 820 of the stream block 810 shown is configured to determine one or more processing parameters (e.g. s, t) for a given stream block based on the training audio signal 802 or a signal derived from the training audio signal and based on a distorted version 801 of the training audio signal (e.g. provided in the adaptation information 821).

[0268] As another optional feature, the neural network associated with a given stream block (e.g. Figure 8 The neural network 820 of the stream block 810 shown is configured to provide one or more parameters of an affine process that is applied to the training audio signal during processing, or to a processed version of the training audio signal during processing, or to a portion of the training audio signal during processing, or to a portion of the processed version of the training audio signal during processing. Thus, the neural network 820 provides information 821 to the stream block 810.

[0269] Optionally, the neural network associated with a given stream block can be configured to determine one or more parameters (s, t) of the affine processing based on a first portion (x1) of the stream block input signal (x) or based on a first portion of the preprocessed stream block input signal (x') and based on a distorted version of the training audio signal. In addition, optionally, the affine processing associated with a given stream block can be configured to apply the determined parameters to a second portion (x2) of the stream block input signal (x) or to a second portion of the preprocessed stream block input signal (x') to obtain an affine processed signal (x2^). In addition, optionally, the first portion (x1) of the stream block input signal (x) or the first portion of the preprocessed stream block input signal (x') and the affine processed signal (x2^) can form a stream block output signal of the given stream block.

[0270] As an optional feature, the apparatus 800 is configured to apply a reversible convolution to a stream block input signal (x) of a given stream block to obtain a pre-processed stream block output signal (x'). This may be performed by the stream block 810, for example, before splitting the input signal into several parts.

[0271] As an optional feature, the apparatus 800 (e.g., the flow block 810) is configured to apply a nonlinear input compression expansion to the training audio signal (x) before processing the training audio signal (x). Thus, as an optional feature, the apparatus is configured to apply a μ-law transform as a nonlinear input compression expansion to the training audio signal (x).

[0272] according to A transformation of can be performed on the training audio signal (x), where sgn() is the sign function and where μ is a parameter defining the compression level.

[0273] In general, as mentioned above, it is noted that, optionally, one or more stream blocks (thus Figure 8 The stream block 810) is configured to convert the training audio signal into a training result signal.

[0274] Optionally, one or more stream blocks are adjusted to transform the training audio signal into a training result signal under the guidance of a distorted version of the training audio signal using affine processing of sample values ​​of the training audio signal or affine processing of sample values ​​of a signal derived from the training audio signal, wherein processing parameters (e.g. s, t) of the affine processing are determined based on sample values ​​of the distorted version of the training audio signal using a neural network.

[0275] In particular, as an optional feature, the flow block 810 is configured to perform a normalization flow process to derive a training result signal from the training audio signal.

[0276] Fig. 9A schematic block diagram of a method for providing a processed audio signal based on an input audio signal according to an embodiment of the second aspect of the present invention is shown. The method 900 includes processing (910) a noise signal or a signal derived from the noise signal using one or more stream blocks to obtain a processed audio signal, applying (930) a depthwise separable convolution to a representation of the input audio signal based on the input audio signal and adapting (920) the processing performed using the one or more stream blocks using a neural network to obtain a preprocessed representation of the input audio signal. In addition, the neural network receives the preprocessed representation of the input audio signal.

[0277] Fig.10 A schematic block diagram of a method for providing a processed audio signal based on an input audio signal according to an embodiment of the third aspect of the present invention is shown. Method 1000 includes processing (1010) a noise signal or a signal derived from the noise signal using one or more stream blocks to obtain a processed audio signal, and adapting (1020) the processing performed using one or more stream blocks based on the input audio signal and using a neural network. In addition, the one or more stream blocks include at least one dual-coupled stream block, wherein the dual-coupled stream block applies (1030) a first affine transformation to a first portion of the input signal to be modified by the dual-coupled stream block, and wherein the dual-coupled stream block applies a second affine transformation to a second portion of the input signal to be modified by the dual-coupled stream block.

[0278] Fig.11 A schematic block diagram of a method for providing a processed audio signal based on an input audio signal according to an embodiment of the first aspect of the invention is shown. The method 1100 comprises processing (1110) a noise signal or a signal derived from the noise signal using one or more stream blocks to obtain a processed audio signal, adapting (1120) the processing performed using the one or more stream blocks based on the input audio signal and using a neural network, and obtaining (1130) a preprocessed representation of the input audio signal using a filter bank, the filter bank comprising a time resolution and / or a frequency resolution adapted to the time resolution and / or frequency resolution of the human auditory system. In addition, the neural network receives the preprocessed representation of the input audio signal.

[0279] Fig.12A schematic block diagram of a method for providing neural network parameters for audio processing according to an embodiment of the second aspect of the present invention is shown. The method 1200 includes processing (1210) a training audio signal or a processed version thereof using one or more stream blocks to obtain a training result signal, adapting (1220) the processing performed using one or more stream blocks based on a distorted version of the training audio signal using a neural network, determining (1230) neural network parameters of the neural network so that the features of the training result audio signal approximate or include predetermined features, and applying (1240) a depthwise separable convolution to a representation of the distorted version of the training audio signal to obtain a preprocessed representation of the distorted version of the training audio signal. In addition, the neural network receives the preprocessed representation of the distorted version of the training audio signal.

[0280] Fig.13 A schematic block diagram of a method for providing neural network parameters for audio processing according to an embodiment of the third aspect of the present invention is shown. Method 1300 includes using one or more stream blocks to process (1310) a training audio signal or a processed version thereof to obtain a training result signal, adapting the processing performed using one or more stream blocks based on a distorted version of the training audio signal and using a neural network, and determining (1320) neural network parameters of the neural network so that the characteristics of the training result audio signal approximate or include predetermined characteristics. In addition, the one or more stream blocks include at least one dual-coupled stream block; wherein the dual-coupled stream block applies (1340) a first affine transformation to a first portion of the input signal to be modified by the dual-coupled stream block, and wherein the dual-coupled stream block applies a second affine transformation to a second portion of the input signal to be modified by the dual-coupled stream block.

[0281] Fig.14 A schematic block diagram of a method for providing neural network parameters for audio processing according to an embodiment of the first aspect of the present invention is shown. The method 1400 includes processing (1410) a training audio signal or a processed version thereof using one or more stream blocks to obtain a training result signal, adapting (1420) the processing performed using the one or more stream blocks based on the distorted version of the training audio signal and using a neural network; determining (1430) neural network parameters of the neural network so that the characteristics of the training result audio signal approximate or include predetermined characteristics, and obtaining (1440) a preprocessed representation of the distorted version of the training audio signal using a filter bank, the filter bank including a time resolution and / or a frequency resolution adapted to the time resolution and / or frequency resolution of the human auditory system. In addition, the neural network receives the preprocessed representation of the distorted version of the training audio signal.

[0282] Further embodiments and aspects

[0283] In the following, further aspects and embodiments according to the present invention will be described, which may be used alone or in combination with any other embodiments disclosed herein.

[0284] In addition, the embodiments disclosed in this section may optionally be supplemented, alone or in combination, by any other features, functions, and details disclosed herein.

[0285] In the following, the concept of improved normalized flow based speech enhancement using an all-pole gammatone filter bank for conditional input representation will be described.

[0286] An overview is provided below.

[0287] In recent years, deep generative models for speech enhancement (SE) have received increasing attention. The goal is to recover clean speech from a signal weakened by background noise. Generative Adversarial Networks (GAN) are the most prominent example, while normalized streams of SE have received less attention, although their potential has not yet been developed. One purpose of an embodiment of the present invention is to further clarify stream-based SE. Based on previous work, structural modifications according to embodiments are proposed and disclosed, which can allow for improving its performance and the efficiency of the training process, as well as research on different conditional input representations according to embodiments of the present invention. Although a common choice in related work, Mel spectromograms have been shown to be insufficient for given scenarios. Alternatively, for example, a new all-pole gammatone filter bank (APG) with high temporal resolution is proposed. Therefore, embodiments of the present invention may include an all-pole gammatone filter bank. Experimental evaluation is performed on the VoiceBank DEMAND dataset and disclosed. Although the results of the computational evaluation metrics show that in some cases, the most advanced GAN-based methods perform best, the perceptual evaluation via the listening test indicates that the normalized flow method (e.g., based on the time domain and APG) proposed in accordance with the embodiments of the present invention performs best, especially at lower signal-to-noise ratios (SNR). On average, the APG output is rated as good quality, which is unmatched by other methods including GAN.

[0288] Index Terms: Speech enhancement, Normalized stream.

[0289] In the following (eg, Section 1 as a concept for improved normalized stream based speech enhancement) an introduction will be provided.

[0290] Speech enhancement (SE) aims to improve the quality of speech that is impaired by interfering background noise [1]. It has a variety of applications, including automatic speech recognition [2], speech coding [3], hearing assistance [4], and broadcasting [5]. SE has been widely studied, and methods based on deep neural networks (DNNs) have largely surpassed traditional techniques such as spectral subtraction [6], Wiener filters [7], or subspace methods [8]. Most commonly, a separation mask is estimated by minimizing a distance metric to extract clean speech components in the time-frequency (TF) domain [9, 10] or a learned subspace

[11] . Nevertheless, in recent years, there has been an increasing interest in generative methods that attempt to characterize the probability distribution of speech signals. The most prominent examples include generative adversarial networks (GANs) [12, 13], variational autoencoders (VAEs)

[14] , autoregressive models

[15] , and diffusion probability models

[16] . GAN-based structures stand out in terms of performance. For example, MetricGAN+

[12] and HiFi-GAN-2

[13] are successors of DNNs that are adversarially trained for SE, respectively. MetricGAN+ optimizes directly on PESQ

[17] or STOI

[18] , reporting high values ​​on the corresponding metrics at output, while HiFi-GAN-2 is pre-trained in a discriminative manner and then adversarially optimized to improve perceptual quality. While the presented results are indeed impressive, in general, GANs are known to be difficult to train in a stable manner and often suffer from mode collapse

[19] .

[0291] Diffusion probabilistic models are a recent example of generative models, where the transformation from Gaussian noise to clean input is learned via a diffusion process. Lu et al.

[16] first applied this approach to SE, recovering clean speech by conditioning the processing on noisy speech. They have shown leading performance in time-domain generative models and show promising generality under mismatch conditions. Nevertheless, sampling from diffusion processes is still quite slow and computationally expensive

[20] . Normalizing Flows (NF)

[21] are another generative modeling technique. They are trained by directly maximizing the likelihood of the data, making them simple and stable to train. Despite their growing success in fields such as computer vision

[22] or speech synthesis

[23] , their application to SE has received little attention. Nugraha et al.

[24] combined NF with VAE to learn deep speech before combining it with the SE algorithm of choice. In contrast, Strauss et al.

[25] used NF to learn a mapping from Gaussian noise to clean speech conditions entirely in the time domain for noisy speech samples. While outperforming the results of other time-domain GAN-based methods, the overall performance evaluated on computational metrics lags behind comparable TF-domain methods.

[0292] For example, based on previous work, one object of the present disclosure is to provide further insights into NF-based SE, for example, to better understand embodiments according to the present invention that include and / or improve NF-based SE. Embodiments according to the present invention improve the original structure, for example, especially through a low complexity or even simple dual coupling scheme, for example, to ensure that (e.g., all) input signals are processed in one stream block. In addition, different input representations of conditional noisy input signals are considered. Therefore, embodiments may include different input representations for conditional noisy input signals and may not be limited to a specific input representation. Our experiments show that although Mel-spectrograms are a common choice for conditional signal representation in related fields (such as neural speech encoders [23, 26]), it is not sufficient for our scenario. Alternatively, a Bark-spaced all-pole gammatone filter bank (APG)

[27] is proposed. Therefore, embodiments may include a Bark-spaced all-pole gammatone filter bank. Similar to Mel, such a design may, for example, utilize a perceptually motivated filter bank, for example to mimic the human auditory system and / or reduce the dimensionality of the filter output, for example compared to a standard Short-Time Fourier Transform (STFT). At the same time, the temporal resolution of a design with such a filter bank according to an embodiment may, for example, be improved, which may optionally overcome the limitations of a standard Mel spectrum. Perceptual evaluation via listening tests indicates that current NF methods based on the time domain and APG according to embodiments (for example) may perform better than state-of-the-art GAN-based methods, for example, especially at lower signal-to-noise ratios, even if this is not reflected by computing evaluation metrics.

[0293] In the following (eg, Section 2 as a concept of improved normalized stream based speech enhancement), normalized stream based speech enhancement will be discussed.

[0294] Let us define the random variable and NF is defined by a differentiable function f that has a differentiable inverse and allows bijective transformations between two random variables

[21] , i.e.,

[0295] x=f(z), z=f -1 (x). (1)

[0296] The reversibility of f ensures that the random variable x is defined by a given probability distribution and can be computed by changes in the variable, i.e.,

[0297] p x (x) = p z(z)|det(J(x))|, (2)

[0298] in is the Jacobian matrix containing all first-order derivatives. Since f is invertible, this also applies to the function f 1：T sequence, that is,

[0299]

[0300] Now let us introduce a single-channel noisy speech signal with a sequence length of N Spoken by clean voice and background noise Add up and you get

[0301] y=x+n. (4)

[0302] Furthermore, z is defined to be sampled from a Gaussian distribution with zero mean and unit variance, i.e.,

[0303]

[0304] The purpose of NF-based SE may now be to characterize the conditional probability distribution p via a DNN with parameters θ x (x|y). Thus, the overall training objective can be described, for example, by maximizing the log-likelihood, i.e., or, for example,

[0305]

[0306] Reverse the learned network, for example, from p z The noise samples sampled in (z) can, for example, be conditioned on a noisy speech utterance and mapped back to the distribution of a clean speech utterance, thereby producing an enhanced speech output.

[0307] In the following, the proposed method according to an embodiment of the present invention is discussed (eg in Section 3 as the concept of improved normalized stream based speech enhancement).

[0308] First (eg, as a first subsection of Section 3), a model structure according to an embodiment is discussed.

[0309] The model used in the experiment is based on

[25] . Embodiments according to the present invention may include any or all of the features disclosed in

[25] alone or in combination. A flow block includes or, for example (see, for example Figure 1) consists of a combination of a reversible 1x1 convolutional layer

[28] and an affine coupling layer

[22] . In a first step, the input signal may be subsampled, for example, by a factor group size, for example, to create a multi-channel input. For example, for a reversible convolutional layer, the input may be split, for example, in two halves, for example, where one part is provided to a subnetwork, for example within a coupling layer, for example, to learn affine transformation parameters for the second half. The transformed signal may be concatenated with the unchanged second part, for example, and may be used, for example, as input to the next block. This operation may be reversible, for example, ensuring that the network is overall reversible, for example, although the subnetwork within the coupling layer that estimates the affine parameters need not be reversible. Similar to Waveglow

[123] , the subnetwork used in embodiments of the present invention may be, for example, a stack of dilated convolutions, for example, with skip connections applied to the input signal, for example, with conditional information introduced by gated activations, for example, as proposed in WaveNet

[29] .

[0310] For example, in order to increase the capacity of each block, a dual coupling scheme (for example, an improved version of the dual coupling scheme of

[30] or for example a dual coupling scheme inspired by

[30] ) can be implemented (or in other words, an embodiment of the present invention may include a dual coupling scheme), for example, where the output of the affine transformation is reused as input to calculate the affine parameters of the second part, i.e. as an example.

[0311]

[0312] Where, as an example, the input x is split into x1 and x2 and s1, t1 and s2 and t2 can be estimated, for example, by the corresponding sub-networks. The outputs can be connected, for example, and is passed to the next stream block, for example. If the order is maintained, the scheme can optionally remain reversible, for example, also when reversing the network.

[0313] Fig.15 A schematic diagram of a dual coupling scheme according to an embodiment of the present invention is shown. For example, before entering the affine coupling layer, the subsampled input x can be, for example, passed through a reversible 1x1 convolution. The conditional input y can, for example, be used as the input of two subnetworks. For example, a stream block (e.g., 110, 210, 310, 510, 610, 710, 720) can include the structure 1500 shown. Therefore, x can be a noise signal (e.g., 102, 202, 302, 502, 602, 702), y can be an input audio signal (e.g., 101, 201, 301, 501, 601, 701), and a neural network (e.g., 120, 220, 320, 520, 620, 712, 714, 722, 724, 820) can include or be a subnetwork 1520.

[0314] Second, (e.g., as the second subsection of Section 3) a discussion of conditional input representation according to an embodiment

[0315] Following the original time domain as a representation of the conditioned signal, experiments with additional variations were performed. Mel-spectrograms are a common choice for conditional representations and work well in neural speech encoders [23, 26]. Embodiments according to the present invention may optionally include a Mel-spectrogram, for example, as disclosed in [23, 26]. For example, the time frame may be upsampled, for example using a transposed convolutional layer as in

[23] , to map the time input dimension. Therefore, embodiments according to the present invention may include upsampled and mapped time frames, for example as disclosed in

[23] . As an alternative, a specially designed complex-valued all-pole gammatone filter bank (APG) is proposed. Therefore, embodiments according to the present invention may trade off the APG, for example, as described below. Inspired by human hearing, the center frequencies of the IIR filters may optionally have a constant distance on the Bark scale, for example, with increasing bandwidth as the frequency increases, which may be, for example, proportional to the Bark bandwidth. Wider filters typically have shorter impulse responses, resulting in increased temporal resolution. To compensate for possible time differences in the filter outputs, the look-ahead of each filter can be implemented, for example, according to its group delay at the center frequency, e.g. scaled with a common factor for all bands. For example, only the amplitudes produced by the filter bank can optionally be further processed by the network. Fig.16 Examples of the magnitude of the impulse responses of selected filter designs (eg, filter designs according to embodiments of the present invention) are shown.

[0316] Fig.16 : A schematic diagram showing an example of the magnitude of the filter response of an all-pole gammatone filter bank (APG) (e.g., filter banks 130, 230, 540) according to an embodiment of the present invention. The filter output may, for example, be delay compensated. For better visibility, only every 10th band is shown (best viewed in color)

[0317] In the following (eg, Section 4 as a concept for improved normalized stream based speech enhancement), an experimental setup according to an embodiment is discussed.

[0318] First, (eg, as the first subsection of Section 4) the datasets considered for the embodiments are discussed.

[0319] For the experiments, we consider the commonly used VoiceBank DEMAND dataset

[31] . It consists of 30 speakers, divided into 28 for training and 2 for testing. The dataset items consist of speech samples from the VoiceBank corpus

[32] contaminated by noise items from the DEMAND database

[33] and artificially generated speech-shaped and noisy noise. These items are mixed with signal-to-noise ratios (SNR) of 0, 5, 10, and 15 dB for training. For testing, SNRs of 2.5, 7.5, 12.5, 17.5 dB are used. In our experiments, SNRs of 2.5, 7.5, 12.5, and 17.5 dB are used. In our experiments, one male and one female speaker are extracted from the training set to construct a development set. All items are resampled to 16 kHz.

[0320] Second, (eg, as a second subsection of Section 4) model configuration according to an embodiment is discussed.

[0321] The models consist of 16 stream blocks connected with group size = 12. The subnetwork has 8 layers of dilated convolutions, which are implemented as depthwise separable convolutions. In contrast to

[25] , the number of output channels of the dilated convolutions is set to 128 and the conditional input layers are replaced by depthwise separable convolutions. This configuration of the model with time domain input has a total of 8.8M parameters, significantly lower than in

[25] . Using a dual coupling scheme may, for example, approximately double the number of parameters, for example, because each block may optionally need to learn two subnetworks. From each training audio file, 1 second long blocks are randomly extracted and used as input to the network. The models are trained with a batch size of 4 and the Adam optimizer (learning rate = 0.001). The learning rate is decayed by a factor of 0.5 if the validation loss does not decrease for 10 consecutive epochs. All models are trained for 200 epochs. Similar to previous work [23, 25], using a lower standard deviation of the sampling distribution in inference experimentally shows slightly better performance. Therefore, the standard deviation is reduced from σ = 1.0 in training to σ = 0.9 in boosting.

[0322] From the Mel spectrum, the FFT parameters are selected as 512 samples for input and window size, with a Hann window and an overlap of 75%. The spectrogram includes 80 bands. The APG is implemented using a 4th order filter and a look-ahead factor of 0.7 is used for group delay compensation. The minimum center frequency is set to 40 Hz, with a total of 80 bands, and the maximum center frequency is just below the Nyquist frequency. Therefore, as an example, an embodiment according to the present invention may include the aforementioned parameters, for example, with a tolerance of + / -5%, or a tolerance of + / -10%, or a tolerance of + / -50%, or a tolerance of + / -100%.

[0323] In the following (eg Section 5 as a concept for improved normalized stream based speech enhancement), embodiments are evaluated.

[0324] We compare the following flow-based systems. The original model

[25] with a single coupled time-domain conditional input is denoted as SE-Flow se , while the proposed dual-coupled version is called SE-Flow. Different input conditions are characterized by the suffixes -Mel and -APG. Two state-of-the-art generative models, MetricGAN+

[12] and CDiffuSE

[16] , are also considered. The samples enhanced by MetricGAN+ are obtained from the model in the speechbrain project. For CDiffuSE, the outputs are kindly provided by the authors of the corresponding paper.

[0325] First, (eg, as the first subsection of Section 5) computing evaluation metrics is discussed.

[0326] These methods are first evaluated using computational metrics. PESQ

[17] (worst: -0.5; best: 4.5) and mean opinion score of the estimated composite measure

[34] (worst: 1; best: 5) are commonly reported in this dataset. For further insight, STOI

[18] (expressed in %; best: 100) and 2f-model scores [35, 26] (worst: 0; best: 100) are also reported.

[0327] Secondly, (e.g. as the second subsection of Section 5) listening tests are discussed.

[0328] The considered methods were also compared via listening tests following the MUSHRA methodology

[37] , under reference conditions and a 3.5 kHz low-pass anchor. Participants were instructed to rate the overall sound quality of the presented items relative to the reference. Test conditions were selected from BUS and CAFE noise settings and only included the most difficult SNR conditions in the test set, i.e., SNRs of 2.5 and 7.5 dB. Each test speaker randomly selected an item of at least 3 seconds from a total of 8 test items. Repeated computational evaluation of the test items confirmed that the selection was not biased towards a particular model. The samples used for this test as well as the input unprocessed signal can be found online2.

[0329] The raw outputs from different systems vary widely in total energy. For an example item, the output composite loudness

[38] can range from -17 to -24 LUFS, while both the noisy input and the reference clean speech are -22.8 LUFS. In addition, different levels of leakage noise are observed after processing. This can make it difficult to assign an overall quality score to the compared systems, as noise suppression tends to be inversely proportional to speech quality. To ensure that a fair comparison of different systems is possible, leakage noise level matching and loudness normalization are performed similar to

[39] . The first background component is obtained by subtracting the enhanced output and the clean reference from the input mix. Then:

[0330] 1. The reference condition was created by mixing the reference clean speech with the corresponding background component with an attenuation factor of 30 dB.

[0331] 2. Determine the speech activity information by thresholding the envelope of the clean reference.

[0332] 3. Determine the integrated loudness of the non-speech portion of the reference condition (gating not activated).

[0333] 4. For each test condition, iteratively obtain the noise attenuation level until the same loudness of the non-speech portion is achieved as in the reference condition, using the same speech activity information collected for the reference condition.

[0334] 5. Each condition was normalized to -23 LUFS (combined loudness, gate not activated).

[0335] The tests were conducted online using the webMUSHRA API, with each participant using their own computer and headphones. The participants were 20 colleagues with different levels of experience in audio research. No results had to be removed according to the MUSHRA post-screening procedure. Note that the test items for CDiffuSE are generated from the original network output, while the numbers reported in the paper include reconstruction with the initial noisy signal. However, this would result in a significant introduction of the initial noise in the enhanced speech part, which would give this method an unfair disadvantage.

[0336] Results and discussion are provided below (eg, Section VI as a concept for improved normalized flow based speech enhancement).

[0337] Fig.17a A table showing an example of experimental results obtained from the VoiceBank-DEMAND test set according to an embodiment of the present invention is shown. The average value with the best result is shown in bold.

[0338] First, (e.g., as the first subsection of Section 6) we discuss the results of computing the metrics

[0339] Fig.17a (Table 1, for example) shows the results of the computational evaluation.

[0340] SE-Flow outperforms its single-coupled counterpart, confirming the benefits of the proposed dual-coupled structure at the expense of a more complex network. SE-Flow with time-conditioned input performs best among the flow-based models. SE-Flow-Mel shows the lowest performance, even worse than the noisy baselines for some metrics. A possible explanation is that the Mel representation of noisy speech utterances is suboptimal for the application at hand, as it does not provide phase information about the input. Comparing SE-Flow-APG with other methods shows that while this model achieves lower results in terms of PESQ and synthetic measures, the results of the 2f model lag only behind the time-domain flow model. MetricGAN+ shows the best results in PESQ and synthetic measures. These results are somewhat expected, as MetricGAN+ directly optimizes PESQ. On STOI, MetricGAN+ shows the best performance together with SE-Flow. CDiffuSE outperforms all flow-based models in terms of PESQ, CSIG, CBAK, and COVL, lagging only behind MetricGAN+. This confirms the results reported in their publications on other time-domain generative models. Regarding the 2-f model, time-domain SE-Flow shows the best performance among all methods.

[0341] The calculated metrics were also evaluated for 7.5 and 2.5 dB SNR conditions. Although the absolute values ​​are significantly different, the main trends and ranking of the methods are similar. Fig.17a Keep the same.

[0342] Secondly, (e.g. as the second subsection of Section 6) the listening test results are discussed.

[0343] Fig.17b : Schematic diagram showing an example of listening test results (20 listeners). Mean and 95% confidence interval (Student's t distribution). Results for different input conditions (7.5dB and 2.5dB SNR) are shown and all items are covered (best viewed in color)

[0344] The results of the listening test are as follows Fig.17bAs shown. The average results of all projects show that SE-Flow-APG performs the best among all methods rated as having good quality on average, and the confidence intervals do not overlap with other methods. SE-Flow follows closely. Despite the high values ​​of the computational evaluation metrics, MetricGAN+ performs worse than SE-Flow, SE-FlowAPG, and CDiffuSE. Inspection of some enhanced samples reveals artifacts in the speech timers, which can partly explain the low results. CDiffuSE ranks third after the two proposed flow-based methods. Inspection of the corresponding samples reveals the low-pass filter-like characteristics of the output samples, which can to some extent explain the results. As also shown by the computational evaluation, SE-Flow-Mel has the worst performance, with scores similar to the low-pass anchor.

[0345] Considering the results grouped by input SNR conditions, SE-Flow-APG performs best in the lowest SNR conditions, while being comparable to SE-Flow at 7.5dB SNR. In fact, the performance of SE-Flow drops dramatically when the SNR drops from 7.5dB to 2.5dB, where the superiority of SE-Flow-APG is evident. Furthermore, MetricGAN+ approximates the good quality range for higher SNR conditions, but drops by 15 MUSHRA points when tested at 2.5dB SNR. CDiffuSe shows greater robustness across SNRs, but has lower overall quality than SE-Flow-APG.

[0346] It is worth emphasizing that the results from listening tests are not consistent with those from computed metrics. In fact, even though computed metrics are very useful in terms of convenience and reproducibility, their correlation with perceived audio quality is often poor

[36] . For this reason, conclusions drawn entirely from computed metrics should be treated with extreme caution, as they can be misleading in terms of perceived quality, as in the reported experiments.

[0347] Conclusions are provided below (eg, Section VII as a concept for improved normalized stream based speech enhancement).

[0348] In the present disclosure, several improvements to the stream-based SE model according to embodiments are introduced and disclosed. Through the dual coupling scheme proposed according to the embodiment, the model can optionally process, for example, the entire input signal in each coupling layer, for example, resulting in higher capacity and performance. Additional experiments consider different representations of the conditional input. Although it is a common choice in the relevant field, the Mel spectrum is not a suitable choice for stream-based SE. As an alternative, the proposed preprocessing of the full-pole gammatone filter bank based on the Bark interval has an increased time resolution, overcoming the problems introduced by Mel. Therefore, embodiments including a full-pole gammatone filter bank and / or preprocessing based on such a filter bank can overcome the problems introduced by Mel. Although in some cases, the results of popular computational metrics may lag behind the most advanced generation models, the results of the listening test show that the stream-based SE of the conditional signal using time domain or gammatone filtering according to the embodiment may have favorable perceptual performance. It can be seen that the proposed method according to the embodiment is not only superior to the compared generation model, but also has strong performance under different signal-to-noise ratio conditions.

[0349] In the following, further aspects and embodiments according to the present invention will be described, which may be used alone or in combination with any other embodiments disclosed herein.

[0350] In addition, the embodiments disclosed in this section may optionally be supplemented, alone or in combination, by any other features, functions, and details disclosed herein.

[0351] In the following, the concept of improved normalized stream based speech enhancement with different input conditions will be described.

[0352] The proposed work is based on previous publications and the embodiments include a method for speech enhancement based on normalized flow.

[0353] The proposed work is based on previous publications and applied patents (PCT / EP2021 / 062076) for normalized stream based speech enhancement.

[0354] 1. Aspects of the invention (features of which may be used alone or in combination with any or all of the embodiments disclosed herein): Use of depthwise separable convolutions as conditional layers.

[0355] In order to introduce the conditional noisy signal into the network, an initial convolution layer can be used, for example, to map the input space to a higher dimension. Initially, this is a standard 1d convolution layer. According to an embodiment of the present invention, the use of a depthwise separable convolution is performed or, for example, proposed, for example to reduce the complexity of this step and / or the number of parameters.

[0356] 2. Aspects of the invention (features of which may be used alone or in combination with any or all of the embodiments disclosed herein): Use of a "dual coupling" scheme within an affine coupling layer.

[0357] For example, in order to ensure the reversibility of a given DNN, an affine coupling layer is used according to an embodiment of the present invention. In this layer, the signal can be, for example, divided into two halves, one of which can, for example, be used to estimate the affine transformation parameters for the second half. At the output, the original first half and the transformed second half can, for example, be connected and forwarded to the next block. The structure of these layers can, for example, keep a part of the signal, for example, unchanged at all times. In order to overcome this, for example and according to an embodiment, the output of the affine transformation can be reused and the affine parameters of the previously unchanged part can be calculated. This may, for example, increase the expressibility (e.g., expressiveness) of the network, for example, at the cost of higher complexity.

[0358] This "double coupling" scheme is inspired by:

[0359] a. Ardizzone et al., “Analysing inverse problems with invertible neural networks,” ICLR, 2019.

[0360] However, the dual coupling scheme of the present invention (eg, as an improved version) is novel for audio applications.

[0361] 3. Aspects of the present invention (features of which may be used alone or in combination with any or all of the embodiments disclosed herein): using an all-pole gammatone filter bank as a pre-processing step for conditioning the input signal.

[0362] In the original work, such as in the previous method, the conditioned input signal is in the time domain. Although it performs well in the previous work, the overall effect is somewhat limited. In other fields, the conditioned signal is often given by another representation, such as a Mel spectrum. As an alternative, according to an embodiment of the present invention, a specially designed complex-valued all-pole gammatone filter bank (APGT) is proposed and / or used. Inspired by human hearing, the center frequency of the IIR filter can have a constant distance on the Bark scale, for example, as the frequency increases, the bandwidth increases, optionally proportional to the Bark bandwidth. Wider filters generally have shorter impulse responses, which may, for example, lead to improved time resolution. For example, in order to compensate for possible time differences in the filter output, for example, the look-ahead of each filter can be implemented according to the embodiment, for example, according to the group delay of each filter at the center frequency, for example, scaling with a common factor for all frequency bands. For example, only the amplitude generated by the filter bank can be further processed, for example, by a network according to an embodiment of the present invention.

[0363] In the following, further aspects and embodiments according to the present invention will be described, which may be used alone or in combination with any other embodiments disclosed herein.

[0364] In addition, the embodiments disclosed in this section may optionally be supplemented, alone or in combination, by any other features, functions, and details disclosed herein.

[0365] In the following, the concept of improved normalized stream based speech enhancement with different input conditions will be further discussed.

[0366] Background according to embodiments of the present invention may include speech enhancement based on normalized streams. However, it should be noted that embodiments according to the present invention may include methods (e.g., methods) for speech enhancement based on normalized streams. Background: PCT / EP2021 / 062076. In particular, embodiments include improved stream-based speech enhancement.

[0367] Fig.18 A schematic visualization of the basic principles of a training process according to an embodiment of the present invention is shown. As shown, a clean speech signal, a training audio signal can be provided to an apparatus according to an embodiment, the apparatus according to the embodiment, for example, including a stream block and a neural network, such as a stream block including a neural network. The processing performed using one or more stream blocks can be adapted based on a distorted version 1820 of the training audio signal 1810 and using a neural network 1830 (e.g., a deep neural network (DNN)). Based on the stream block, a training result audio signal 1840 can be provided. The neural network can be trained to include neural network parameters to best adapt to the processing performed using the stream block so that the training result audio signal approximates or includes predetermined features, such as noise features.

[0368] Embodiments may include any or all of the following features:

[0369] Learning a mapping from clean speech to white noise conditioned on its noisy counterpart; adapted models from speech synthesis; working in the time domain.

[0370] As an example, embodiments may therefore be performed in the time domain.

[0371] Fig.19 A schematic visualization of the basic principles of the training process together with the enhancement process according to an embodiment of the present invention is shown. Based on the training, the transformation performed in the corresponding stream block can be reversed so that the noise signal 1910 is transformed into a processed audio signal (enhanced speech signal) based on the adaptation of the processing in the corresponding stream block using the neural network and the input audio signal 1920.

[0372] Fig. 20Schematic visualization of an improvement (eg, a first improvement) according to aspects of the present invention is shown. Fig. 20 The features shown can be used individually or in combination. Fig. 20 As shown, preprocessing including depth-wise separable convolution may be performed based on a distorted version 2020 of the training audio signal 2010. Fig. 20 , the corresponding training process and the training result audio signal 2040 are shown. Also, it should be noted that Fig. 20 The features shown in can be used alone or in combination.

[0373] Fig.21 and Fig. 22 A schematic visualization of a refinement (eg, the second refinement) according to aspects of the present invention is shown.

[0374] like Fig.21 As shown, embodiments may include a single coupling method. An input signal (e.g., x) of a stream block may be split, wherein a first portion is provided to a neural network or subnetwork, based on which parameters of an affine transformation may be provided, the affine transformation processing a second portion of the input signal. The processed second portion and the first portion may thus be connected as a result signal.

[0375] Therefore, a portion of the input signal may remain unchanged, for example substantially unchanged. Fig.21 ,It is noted that the features can be used individually or in combination.

[0376] In contrast, Fig. 22 A schematic diagram of a dual coupling concept according to an embodiment of the present invention is shown. As mentioned above, for example, the entire signal can be changed at once. Therefore, the capacity of the network can be increased.

[0377] Inspired by: L. Ardizzone et al., “Analyzing inverse problems withinvertible neural networks,” ICLR, 2019; but without audio applications.

[0378] Likewise, about Fig. 22 ,It is noted that the features can be used individually or in combination.

[0379] Fig.23 FIG. 2 shows a schematic visualization of the principle of improvement of the training process together with the enhancement process according to an embodiment of the present invention. Fig.23 As shown, Fig.19 By comparison, as an improvement (eg, the third improvement), a filter bank 2310, such as an all-pole gammatone filter bank, may be implemented. Fig.23,It is noted that the features can be used individually or in combination.

[0380] Thus, in summary, for example summarizing improvements according to embodiments, embodiments may include filter banks, for example in the form of all-pole gammatone filter banks, dual coupling schemes and / or conditional layers, using depthwise separable convolutions.

[0381] Embodiments may include any or all of the above features.

[0382] For more information on calculating metrics, see Fig.17a For CDiffuSE, refer to

[15] , and for MetricGAN+, refer to

[16] . MetricGAN+: Research Center for Information Technology Innovation

[0383] Ohio State, USA

[0384] MILA QuebecAI institute,Canada

[0385] CDiffuSE:Carnegie Mellon University,USA

[0386] Research Center for Information Technology Innovation

[0387] -> The latest progress in generative speech enhancement technology

[0388] For examples of listening tests, see Fig.24 .

[0389] Furthermore, possible applications of the embodiment include: various audio objects of MPEG H (an application that divides legacy content into various components), and dialogue enhancement.

[0390] Inventions or improvements over previous versions (e.g., advantages of embodiments) include, inter alia: an APGT filter bank as a conditional signal representation; a dual coupling scheme for improved expressiveness of normalized stream-based speech enhancement; and conditional layers as depthwise separable convolutions.

[0391] In other words, these embodiments include improvements on methods related to stream-based neural networks for speech enhancement in the time domain: first, a dual coupling method is used so that all parts can be processed in one step. Second, an all-pole gammatone filter bank is used to transform the conditioned noisy signal while maintaining the temporal resolution. Third, a depthwise separable convolutional network layer is used as a step between the filter bank transformation and the introduction into the neural network in order to reduce complexity.

[0392] Listening tests have demonstrated improved quality compared to similar methods.

[0393] Note that embodiments may include any or all of these features.

[0394] In general, it is noted that embodiments include or relate to normalized stream based speech enhancement using improved input conditions.

[0395] Implementation alternatives:

[0396] Although some aspects are described herein in the context of an apparatus, it is apparent that these aspects also represent a description of a corresponding method, wherein a block or device corresponds to a method step or a feature of a method step. Similarly, the aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0397] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0398] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0399] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

[0400] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0401] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0402] Therefore, another embodiment of the inventive method is a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.

[0403] Therefore, another embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection (eg via the Internet).

[0404] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0405] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0406] Another embodiment according to the invention comprises an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, comprise a file server for transmitting the computer program to the receiver.

[0407] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may be used in conjunction with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0408] The devices described herein may be implemented using hardware devices, or using computers, or using a combination of hardware devices and computers.

[0409] The apparatus described herein or any component of the apparatus described herein may be implemented at least in part by hardware and / or software.

[0410] The methods described herein may be performed using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0411] Any components of the methods described herein or the apparatus described herein may be performed at least in part by hardware and / or software.

[0412] The above embodiments are only used to illustrate the principles of the present invention. It is to be understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the pending patent claims and not by the specific details presented by the description and explanation of the embodiments herein.

[0413] References

[0414] [1] P. Loizou, Speech Enhancement: Theory and Practice, 2nd ed. CRC Press, 2013.

[0415] [2] A.Pandey, C.Liu, Y.Wang and Y.Saraf, "Dual Application of SpeechEnhancement for Automatic Speech Recognition," in 2021IEEE Spoken Language Technology Workshop (SLT), 2021, pp.223–228.

[0416] [3] Z. Zhao, H. Liu and T. Fingscheidt, "Convolutional Neural Networks toEnhance Coded Speech," IEEE Trans.Audio,Speech,Lang.Process., vol.27, no.4, pp.663–678, 2019.

[0417] [4] G.Park, W.Cho, K.-S.Kim and S.Lee, "Speech Enhancement for Hearing Aidswith Deep Learning on Environmental Noises," Applied Sciences, vol.10, no.17, 2020.

[0418] [5] M. Torcoli, C. Simon, J. Paulus et al., “Dialog+ in Broadcasting: First Field Tests using Deep-Learning Based Dialogue Enhancement,” in Int. Broadcast. Conv. (IBC) Technical Papers, 2021.

[0419] [6] H. Gustafsson, S. Nordholm and I. Claesson, “Spectral Subtraction Using Reduced Delay Convolution and Adaptive Averaging,” IEEE Trans. Speech Audio Process., vol. 9, no. 8, pp. 799–807, 2001.

[0420] [7] J.C., J. Benesty, Y.H. and S. Doclo, “New Insights into the Noise Reduction Wiener Filter,” IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 4, pp. 1218–1234, 2006.

[0421] [8] M. Klein and P. Kabal, “Signal Subspace Speech Enhancement with Perceptual Post-Filtering,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2002, pp. 537–540.

[0422] [9] Y. Koizumi, S. Karita, S. Wisdom et al., “DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancement.”

[0423]

[10] D. Wang and J. Chen, “Supervised Speech Separation Based on Deep Learning: An Overview,” IEEE Trans. Audio, Speech, Lang. Process., vol. 26, no. 10, pp. 1702–1726, 2018.

[0424]

[11] Y. Y. Yi Luo and N. Mesgarani, “Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation,” IEEE Trans. Audio, Speech, Lang. Process., vol. 27, pp. 1256–1266, 2019.

[0425]

[12] S.-W. Fu, C. Yu, T.-A. Hsieh et al., “MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement,” in Proc. Interspeech Conf., 2021, pp. 201–205.

[0426]

[13] J. Su, Z. Jin and A. Finkelstein, “HiFi-GAN-2: Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2021, pp. 166–170.

[0427]

[14] S. Leglaive, X. Alameda-Pineda, L. Girin and R. Horaud, “A Recurrent Variational Autoencoder for Speech Enhancement,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 371–375.

[0428]

[15] K. Qian, Y. Zhang, S. Chang et al., “Speech Enhancement Using Bayesian Wavenet,” in Proc. Interspeech Conf., 2017, pp. 2013–2017.

[0429]

[16] Y.-J. Lu, Z.-Q. Wang, S. Watanabe et al., “Conditional Diffusion Probabilistic Model for Speech Enhancement,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2022 (to appear).

[0430]

[17] International Telecommunication Union, “Recommendation ITU–T P.862 Perceptual evaluation of speech quality (PESQ), and objective method for end-to-end speech quality assessment of narrowband telephone band and wideband digital codes,” 2000.

[0431]

[18] C.H. Taal, R.C. Hendriks, R. Heusdens and J. Jensen, “A Short-Time Objective Intelligibility Measure for Time-Frequency Weighted Noisy Speech,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2010, pp. 4214–4217.

[0432]

[19] L. Metz, B. Poole, D. Pfau and J. Sohl-Dickstein, “Unrolled Generative Adversarial Networks,” in 5th International Conference on Learning Representations, ICLR 2017, 2017.

[0433]

[20] J. Song, C. Meng and S. Ermon, “Denoising Diffusion Implicit Models,” in International Conference on Learning Representations, 2021.

[0434]

[21] G. Papamakarios, E. Nalisnick, D. Rezende et al., “Normalizing Flows for Probabilistic Modeling and Inference,” Journal of Machine Learning Research, vol. 22, no. 57, pp. 1–64, 2021.

[0435]

[22] L. Dinh, J. Sohl-Dickstein and S. Bengio, “Density Estimation Using Real NVP,” in 5th Int. Conf. on Learning Representations, ICLR, 2017.

[0436]

[23] R. Prenger, R. Valle and B. Catanzaro, “Waveglow: A Flow-based Generative Network for Speech Synthesis,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 3617–3621.

[0437]

[24] A. A. Nugraha, K. Sekiguchi and K. Yoshii, “A Flow-Based Deep Latent Variable Model for Speech Spectrogram Modeling and Enhancement,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1104–1117, 2020.

[0438]

[25] M. Strauss and B. Edler, “A Flow-Based Neural Network for Time Domain Speech Enhancement,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 5754–5758.

[0439]

[26] A. Mustafa, N. Pia and G. Fuchs, “Stylemelgan: An efficient high-fidelity adversarial vocoder with temporal adaptive normalization,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 6034–6038.

[0440]

[27] R. F. Lyon, “All-pole models of auditory filtering,” in in Diversity in Auditory Mechanics, 1997, pp. 205–211.

[0441]

[28] D. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” in Advances in Neural Information Processing Systems 31, 2018, pp. 10215–10224.

[0442]

[29] A. van den Oord, S. Dieleman, H. Zen et al., “Wavenet: A generative model for raw audio,” in arXiv, 2016. [Online]. Available: https: / / arxiv.org / abs / 1609.03499

[0443]

[30] L. Ardizzone, J. Kruse, C. Rother and U. Kothe, “Analyzing Inverse Problems with Invertible Neural Networks,” in 7th Int. Conf. on Learning Representations, ICLR, 2019.

[0444]

[31] C. Valentini Botinhao, X. Wang, S. Takaki and J. Yamagishi, “Speech Enhancement for a Noise-Robust Text-to-Speech Synthesis System using Deep Recurrent Neural Networks,” in Proc. Interspeech Conf., 2016, pp. 352–356.

[0445]

[32] C. Veaux, J. Yamagishi and S. King, “The Voice Bank Corpus: Design, collection and data analysis of a large regional accent speech database,” in Int. Conf. Oriental COCOSDA held jointly with the Conf. on Asian Spoken Language Research and Evaluation (O-COCOSDA / CASLRE), 2013, pp. 1–4.

[0446]

[33] J. Thiemann, N. Ito and E. Vincent, “The Diverse Environments Multi-channel Acoustic Noise Database (DEMAND): A database of multichannel environmental noise recordings,” Proc. of Meetings on Acoustics, vol. 19, no. 1, p. 035081, 2013.

[0447]

[34] Y. Hu and P. Loizou, “Evaluation of Objective Quality Measures for Speech Enhancement,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, no. 1, pp. 229–238, 2008.

[0448]

[35] T. Kastner and J. Herre, “An Efficient Model for Estimating Subjective Quality of Separated Audio Source Signals,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2019, pp. 95–99.

[0449]

[36] M. Torcoli, T. Kastner, and J. Herre, “Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of their Application Domain Dependence,” IEEE Trans. Audio, Speech, Lang. Process., vol. 29, pp. 1530–1541, 2021.

[0450]

[37] International Telecommunication Union, “Recommendation ITU–R BS.1534-3 Method for the subjective assessment of intermediate quality level of audio systems,” 2015.

[0451]

[38] ——, “Recommendation ITU–R BS.1770-4 Algorithms to measure audio programme loudness and true-peak audio level,” 2015.

[0452]

[39] M. Strauss, J. Paulus, M. Torcoli, and B. Edler, “A Hands On Comparison of DNNs for Dialog Separation Using Transfer Learning from Music Source Separation,” in Proc. Interspeech Conf., 2021, pp. 3900–3904。

Claims

1. An apparatus (100, 200, 300, 500, 600, 700) for providing a processed audio signal (103, 203, 303, 503, 603, 703) based on an input audio signal (101, 201, 301, 501, 601, 701), wherein the apparatus is configured to process a noise signal (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal using one or more stream blocks to obtain the processed audio signal, wherein the apparatus is configured to adapt processing performed using the one or more stream blocks (110, 210, 310, 510, 610, 710, 720) based on the input audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the apparatus is configured to obtain a preprocessed representation (131, 231, 541, 731, 831) of the input audio signal using a filter bank (130, 230, 540, 830), the filter bank (130, 230, 540, 830) comprising a time resolution and / or a frequency resolution adapted to a time resolution and / or a frequency resolution of a human auditory system; and wherein the neural network is configured to receive the pre-processed representation of the input audio signal.

2. The device (100, 200, 300, 500, 600, 700) according to claim 1, The time resolution of the filter bank (130, 230, 540, 830) and the frequency resolution of the filter bank approximate the time resolution and frequency resolution of the human auditory system.

3. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 2, The filters of the filter bank (130, 230, 540, 830) are infinite impulse response filters.

4. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 3, in, The filter bank (130, 230, 540, 830) is an all-pole gammatone filter bank (230).

5. An apparatus (100, 200, 300, 500, 600, 700) for providing a processed audio signal (103, 203, 303, 503, 603, 703) based on an input audio signal (101, 201, 301, 501, 601, 701), wherein the apparatus (100, 200, 300, 500, 600, 700) is configured to process a noise signal (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal using one or more stream blocks (110, 210, 310, 510, 610, 710, 720) to obtain the processed audio signal, wherein the apparatus is configured to adapt processing performed using the one or more stream blocks based on the input audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the apparatus is configured to use an all-pole gammatone filter bank (230) to obtain a preprocessed representation (231) of the input audio signal; and wherein the neural network is configured to receive the pre-processed representation of the input audio signal.

6. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 5, wherein the all-pole gammatone filter bank (230) is configured to obtain a plurality of channel signals associated with a plurality of frequency bands; wherein the width of the frequency band increases monotonically with the increase of the center frequency of each of the frequency bands, and / or wherein the width of the frequency band is adapted according to a psychoacoustic model, and / or Therein the center frequencies of the frequency bands are adapted according to a psychoacoustic model.

7. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 6, wherein the all-pole gammatone filter bank (230) comprises a plurality of filters; and The center frequency of the filter comprises a constant distance on the Bark scale, wherein the bandwidth increases with increasing frequency.

8. The device (100, 200, 300, 500, 600, 700) according to claim 1, 6 or 7, The all-pole gammatone filter bank (230) is configured to at least partially compensate for different group delays between different filters.

9. The device (100, 200, 300, 500, 600, 700) according to claim 1 or 6 or 7 or 8, The transfer function of the all-pole gammatone filter bank (230) does not include any finite zeros.

10. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 9, The imaginary parts of the poles of the transfer function of the all-pole gammatone filter bank (230) all have the same sign.

11. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 10, The all-pole gamma-tone filter bank (230) is a complex all-pole gamma-tone filter bank.

12. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 11, One or more poles of the transfer function of the all-pole gammatone filter bank (230) coincide.

13. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 12, wherein the apparatus is configured to obtain the preprocessed representation of the input audio signal based on the amplitude of the output signal of the all-pole gammatone filter bank (230); and / or The device is configured to ignore phase information of the output signal of the all-pole gammatone filter bank.

14. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 13, The all-pole gammatone filter bank (230) is configured to provide 20 to 100 output signals.

15. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 14, The apparatus is configured to apply a plurality of convolutions to a set of output values ​​of the all-pole gammatone filter bank (230) or to a set of amplitude values ​​derived from the output values ​​of the all-pole gammatone filter bank to obtain input values ​​of the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820).

16. The device (100, 200, 300, 500, 600, 700) according to any one of claims 1 to 15, The apparatus is configured to apply a depthwise separable convolution to a set of output values ​​(231) of the all-pole gamma-pass filter bank (230) or to a set of amplitude values ​​derived from the output values ​​of the all-pole gamma-pass filter bank to obtain input values ​​of the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820).

17. An apparatus (100, 200, 300, 500, 600, 700) for providing a processed audio signal (103, 203, 303, 503, 603, 703) based on an input audio signal (101, 201, 301, 501, 601, 701), wherein the apparatus is configured to process a noise signal (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal using one or more stream blocks (110, 210, 310, 510, 610, 710, 720) to obtain the processed audio signal, wherein the apparatus is configured to adapt processing performed using the one or more stream blocks based on the input audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the apparatus is configured to apply a depthwise separable convolution to a representation of the input audio signal to derive a preprocessed representation of the input audio signal (251, 331, 531, 841); wherein the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) is configured to receive the pre-processed representation of the input audio signal.

18. The device (100, 200, 300, 500, 600, 700) according to claim 17, Wherein the depth-wise separable convolution is configured to perform temporal convolution and convolution in the frequency direction.

19. The device according to claim 17 or claim 18, wherein the apparatus is configured to obtain a representation of the input audio signal using an all-pole gammatone filter bank (230), and The apparatus is configured to apply the depthwise separable convolution to the representation (231) of the input audio signal obtained using the all-pole gammatone filter bank.

20. The device (100, 200, 300, 500, 600, 700) according to claim 19, Wherein the representation (231) of the input audio signal obtained using the all-pole gammatone filter bank comprises a plurality of sub-band signals.

21. The device (100, 200, 300, 500, 600, 700) according to claim 20, The apparatus is configured to apply different convolutions to the plurality of sub-band signals to obtain an input signal for the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820).

22. The device (100, 200, 300, 500, 600, 700) according to any one of claims 17 to 21, wherein the apparatus is configured to apply respective time convolutions to a plurality of signals representing the input audio signal (101, 201, 301, 501, 601, 701) to obtain a plurality of time convolved signal values; and The apparatus is configured to apply multiple convolutions in frequency to a given set of time convolution signal values ​​to obtain multiple input values ​​for the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820).

23. The device (100, 200, 300, 500, 600, 700) according to any one of claims 17 to 22, The apparatus is configured to apply the depthwise separable convolution to a representation of the input audio signal (101, 201, 301, 501, 601, 701) to map the input space to a higher dimension.

24. The device (100, 200, 300, 500, 600, 700) according to any one of claims 17 to 23, The apparatus is configured to perform a plurality of convolutions in frequency based on the same set of result values ​​of respective temporal convolutions, wherein the respective temporal convolutions are respectively performed based on signals of a frequency domain representation of the input audio signal.

25. The device (100, 200, 300, 500, 600, 700) according to any one of claims 17 to 24, wherein the one or more flow blocks (110, 210, 310, 510, 610, 710, 720) include at least one dual-coupled flow block (510, 610, 710, 720); wherein the dual coupled stream block is configured to apply a first affine transformation (512, 612, 716, 726) to a first portion (502a, 602a, 702a, 731a) of an input signal to be modified by the dual coupled stream block, and Wherein the dual coupled stream block is configured to apply a second affine transformation (514, 614, 718, 728) to a second portion (502b, 602b, 702b, 731b) of the input signal to be modified by the dual coupled stream block.

26. The device (100, 200, 300, 500, 600, 700) according to any one of claims 17 to 25, wherein the apparatus is configured to obtain a preprocessed representation of the input audio signal (131, 231, 541, 731, 831) using a filter bank (130, 230, 540, 830), the filter bank (130, 230, 540, 830) comprising a time resolution and / or a frequency resolution adapted to a time resolution and / or a frequency resolution of a human auditory system; and wherein the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) is configured to receive the pre-processed representation of the input audio signal.

27. The apparatus (100, 200, 300, 500, 600, 700) according to any one of claims 17 to 26, wherein the apparatus is configured to obtain the preprocessed representation of the input audio signal using an all-pole gammatone filter bank (230); and wherein the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) is configured to receive the pre-processed representation of the input audio signal.

28. An apparatus (100, 200, 300, 500, 600, 700) for providing a processed audio signal (103, 203, 303, 503, 603, 703) based on an input audio signal (101, 201, 301, 501, 601, 701), wherein the apparatus is configured to process a noise signal (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal using one or more stream blocks to obtain the processed audio signal, wherein the apparatus is configured to adapt processing performed using the one or more stream blocks (110, 210, 310, 510, 610, 710, 720) based on the input audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the one or more flow blocks include at least one dual-coupled flow block (510, 610, 710, 720); wherein the dual coupled stream block is configured to apply a first affine transformation (512, 612, 716, 726) to a first portion (502a, 602a, 702a, 731a) of an input signal to be modified by the dual coupled stream block, and Wherein the dual coupled stream block is configured to apply a second affine transformation (514, 614, 718, 728) to a second portion (502b, 602b, 702b, 731b) of the input signal to be modified by the dual coupled stream block.

29. The device (100, 200, 300, 500, 600, 700) according to claim 28, wherein the apparatus is configured to adapt the processing to be performed by the first affine transform (512, 612, 716, 726) of the dual coupled stream block (510, 610, 710, 720) using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) based on the input signal of the second affine transform (514, 614, 718, 728), and Wherein the apparatus is configured to adapt, using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820), processing to be performed by the second affine transformation of the dual coupled stream block based on the first portion (502a, 602a, 702a, 731a) of the input signal to be modified by the dual coupled stream block.

30. The device (100, 200, 300, 500, 600, 700) according to any one of claims 28 to 29, wherein the apparatus is configured to apply processing using a sequence of a plurality of dual-coupled flow blocks (510, 610, 710, 720), Wherein the apparatus is configured to apply a reversible convolution to obtain an input signal of a second dual-coupled stream block based on an output signal of a previous first dual-coupled stream block.

31. The device (100, 200, 300, 500, 600, 700) according to any one of claims 28 to 30, wherein the dual coupled stream block (510, 610, 710, 720) is configured to separate the input signal of the dual coupled stream block to obtain the first part (502a, 602a, 702a, 731a) of the input signal and the second part (502b, 602b, 702b, 731b) of the input signal, and apply respective affine transformations to the first part of the input signal and the second part of the input signal.

32. The device (100, 200, 300, 500, 600, 700) according to any one of claims 28 to 31, The device is configured to connect the output signal of the first affine transformation (512, 612, 716, 726) and the output signal of the second affine transformation (514, 614, 718, 728) to obtain the output signal of the dual coupled stream block (510, 610, 710, 720).

33. The device (100, 200, 300, 500, 600, 700) according to any one of claims 28 to 32, The apparatus is configured to use the second portion (502b, 602b, 702b, 731b) of the input signal as an input signal of a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) for determining transformation parameters of the first affine transformation (512, 612, 716, 726) and as an input signal of the second affine transformation (514, 614, 718, 728).

34. The device (100, 200, 300, 500, 600, 700) according to any one of claims 28 to 33, The apparatus is configured to use an output signal of the first affine transformation (512, 612, 716, 726) as an input signal of a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) for determining transformation parameters of the second affine transformation (514, 614, 718, 728).

35. The device (100, 200, 300, 500, 600, 700) according to any one of claims 28 to 34, wherein the dual coupled stream block (510, 610, 710, 720) is configured to split the input signal to be modified by the dual coupled stream block into two halves, and Wherein the dual coupled stream block is configured to use a second half of the input signal to be modified by the dual coupled stream block to estimate parameters of an affine transformation to be applied to a first half of the input signal to be modified by the dual coupled stream block.

36. The device (100, 200, 300, 500, 600, 700) according to any one of claims 28 to 35, The dual coupled stream block (510, 610, 710, 720) is configured to modify only the signal of the first part (502a, 602a, 702a, 731a) of the input signal to be modified by the dual coupled stream block in a first affine transformation, and to modify only the signal of the second part (502b, 602b, 702b, 731b) of the input signal to be modified by the dual coupled stream block in a second affine transformation (514, 614, 718, 728).

37. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, The input audio signal (101, 201, 301, 501, 601, 701) is represented by a set of time-domain audio samples.

38. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, wherein a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) associated with a given stream block (110, 210, 310, 510, 610, 710, 720) of the one or more stream blocks is configured to determine one or more processing parameters of the given stream block based on a noise signal (z) (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal and based on the input audio signal (y) (101, 201, 301, 501, 601, 701).

39. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, wherein a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) associated with a given stream block (110, 210, 310, 510, 610, 710, 720) is configured to provide one or more parameters of an affine process applied to the noise signal (102, 202, 302, 502, 602, 702) during the processing, or to a processed version of the noise signal during the processing, or to a portion of the noise signal during the processing, or to a portion of a processed version of the noise signal during the processing.

40. The device according to claim 39, characterized in that wherein the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) associated with the given stream block (110, 210, 310, 510, 610, 710, 720) is configured to determine one or more parameters of the affine processing based on a first part (z1) of the stream block input signal (z) and based on the input audio signal (y) (101, 201, 301, 501, 601, 701), and wherein the affine processing associated with the given stream block is configured to apply the determined parameters to a second portion (z2) of the stream block input signal (z) to obtain an affine processed signal (z2^); and Wherein the first part (z1) of the stream block input signal (z) and the affine processed signal (z2^) form the stream block output signal of the given stream block.

41. The device (100, 200, 300, 500, 600, 700) according to claim 40, wherein the apparatus is configured to apply a reversible convolution to a stream block (110, 210, 310, 510, 610, 710, 720) output signal (z new ) to obtain the processed stream block output signal (z' new ).

42. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, Wherein the apparatus is configured to apply a non-linear expansion to the processed audio signal.

43. The device according to claim 42, characterized in that The apparatus is configured to apply an inverse μ-law transform as the nonlinear expansion to the processed audio signal (x̂).

44. The apparatus (100, 200, 300, 500, 600, 700) according to any one of claims 42 or 43, wherein the apparatus is configured to The transformation of is applied to the processed audio signal (x^), Where sgn() is the sign function; μ is a parameter that defines the level of expansion.

45. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, wherein the neural network parameters (803) of the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) for processing the noise signal (102, 202, 302, 502, 602, 702) or the signal derived from the noise signal are obtained using processing of a training audio signal (802) or a processed version thereof in one or more training stream blocks to obtain a training result signal, wherein the processing of the training audio signal or the processed version thereof using the one or more training stream blocks (810) is based on a distorted version (801) of the training audio signal and adapted using the neural network, and The neural network parameters of the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) are determined so that the characteristics of the training result audio signal (811) are similar to or include predetermined characteristics.

46. ​​The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, wherein the apparatus is configured to provide neural network parameters (803) of the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) for processing the noise signal (102, 202, 302, 502, 602, 702) or the signal derived from the noise signal, wherein the apparatus is configured to process the training audio signal (802) or a processed version thereof using the one or more stream blocks to obtain a training result signal (811), and wherein the apparatus is configured to adapt the processing of the training audio signal or the processed version thereof performed using the one or more stream blocks based on the distorted version (801) of the training audio signal and using the neural network, and The device is configured to determine the neural network parameters of the neural network so that the characteristics of the training result audio signal are similar to or include predetermined characteristics.

47. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, wherein the apparatus comprises means for providing neural network parameters (803), wherein the means for providing neural network parameters is configured to provide neural network parameters of the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) for processing the noise signal (102, 202, 302, 502, 602, 702) or the signal derived from the noise signal, wherein the means for providing neural network parameters is configured to process the training audio signal (802) or a processed version thereof using one or more training stream blocks to obtain a training result signal (811), and wherein the means for providing neural network parameters is configured to adapt the processing of the training audio signal or the processed version thereof performed using the one or more stream blocks based on the distorted version of the training audio signal and using the neural network; The device is configured to determine the neural network parameters of the neural network so that the characteristics of the training result audio signal are similar to or include predetermined characteristics.

48. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, The one or more stream blocks are configured to synthesize the processed audio signal (103, 203, 303, 503, 603, 703) based on the noise signal (102, 202, 302, 502, 602, 702) under the guidance of the input audio signal (101, 201, 301, 501, 601, 701).

49. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, wherein the one or more stream blocks (110, 210, 310, 510, 610, 710, 720) are configured to synthesize the processed audio signal (103, 203, 303, 503, 603, 703) based on the noise signal (102, 202, 302, 502, 602, 702) using the affine processing of sample values ​​of the noise signal or of a signal derived from the noise signal, under the guidance of the input audio signal (101, 201, 301, 501, 601, 701), The processing parameters (e.g., s, t) of the affine processing are determined based on sample values ​​of the input audio signal using the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820).

50. The device (100, 200, 300, 500, 600, 700) according to any of the preceding claims, The apparatus is configured to perform normalized stream processing to derive a processed audio signal from the noise signal.

51. An apparatus (800) for providing neural network parameters (803) for audio processing, wherein the apparatus is configured to process the training audio signal (802) or a processed version thereof using one or more stream blocks (810) to obtain a training result signal (811), wherein the apparatus is configured to adapt processing performed using the one or more stream blocks based on a distorted version (801) of the training audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the apparatus is configured to determine the neural network parameters of the neural network so that the characteristics of the training result audio signal (811) approximate or include predetermined characteristics, wherein the apparatus is configured to apply a depthwise separable convolution to the representation of the distorted version of the training audio signal to derive a preprocessed representation of the distorted version of the training audio signal; wherein the neural network is configured to receive the pre-processed representation (831, 841) of the distorted version of the training audio signal.

52. An apparatus (800) for providing neural network parameters (803) for audio processing, wherein the apparatus is configured to process the training audio signal (802) or a processed version thereof using one or more stream blocks (810) to obtain a training result signal (811), wherein the apparatus is configured to adapt processing performed using the one or more stream blocks based on a distorted version (801) of the training audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the apparatus is configured to determine the neural network parameters of the neural network so that the characteristics of the training result audio signal approximate or include the predetermined characteristics, wherein the one or more flow blocks include at least one dual-coupled flow block (510, 610, 710, 720); wherein the dual coupled stream block is configured to apply a first affine transformation (512, 612, 716, 726) to a first portion (502a, 602a, 702a, 731a) of an input signal to be modified by the dual coupled stream block, and Wherein the dual coupled stream block is configured to apply a second affine transformation (514, 614, 718, 728) to a second portion (502b, 602b, 702b, 731b) of the input signal to be modified by the dual coupled stream block.

53. An apparatus (800) for providing neural network parameters (803) for audio processing, wherein the apparatus is configured to process the training audio signal (802) or a processed version thereof using one or more stream blocks to obtain a training result signal (811), wherein the apparatus is configured to adapt processing performed using the one or more stream blocks (810) based on a distorted version (801) of the training audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the apparatus is configured to determine neural network parameters of the neural network so that a feature of the training result audio signal approximates or includes a predetermined feature; wherein the apparatus is configured to obtain a preprocessed representation (831) of the distorted version of the training audio signal using a filter bank (130, 230, 540, 830), the filter bank (130, 230, 540, 830) comprising a time resolution and / or a frequency resolution adapted to the time resolution and / or the frequency resolution of the human auditory system; and wherein the neural network is configured to receive the pre-processed representation of the distorted version of the training audio signal.

54. The device (800) according to any one of claims 51 to 53, wherein the apparatus is configured to evaluate a cost function according to characteristics of the obtained training result signal (811), and The apparatus is configured to determine neural network parameters to reduce or minimize a cost defined by the cost function.

55. The apparatus (800) according to any one of claims 51 to 54, wherein the training audio signal (802) and / or the distorted version (801) of the training audio signal is represented by a set of time-domain audio samples.

56. An apparatus (800) according to any one of claims 51 to 55, wherein the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) associated with a given stream block of the one or more stream blocks is configured to determine one or more processing parameters (e.g., s, t) of the given stream block based on the training audio signal or a signal derived from the training audio signal and based on the distorted version of the training audio signal.

57. An apparatus (800) according to any one of claims 51 to 56, wherein the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) associated with a given stream block is configured to provide one or more parameters of an affine processing, wherein the affine processing is applied to the training audio signal during the processing, or to a processed version of the training audio signal during the processing, or to a portion of the training audio signal during the processing, or to a portion of the processed version of the training audio signal during the processing.

58. The apparatus (800) of claim 57, wherein the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820) associated with the given stream block is configured to determine one or more parameters (s, t) of the affine processing based on a first part (x1) of a stream block input signal (x) or based on a first part of a pre-processed stream block input signal (x') and based on the distorted version of the training audio signal, and wherein the affine processing associated with the given stream block is configured to apply the determined parameters to a second portion (x2) of the stream block input signal (x) or to a second portion of the pre-processed stream block input signal (x') to obtain an affine processed signal (x2^); and Wherein the first part (x1) of the stream block input signal (x) or the first part of the pre-processed stream block input signal (x') and the affine processed signal (x2^) form the stream block output signal of the given stream block.

59. The apparatus (800) of claim 58, wherein the apparatus is configured to apply a reversible convolution to the stream block input signal (x) of the given stream block to obtain the pre-processed stream block input signal (x').

60. The apparatus (800) according to any one of claims 51 to 59, wherein the apparatus is configured to apply non-linear input compression and expansion to the training audio signal (x) before processing the training audio signal (x).

61. The apparatus (800) according to claim 60, wherein the apparatus is configured to apply a μ-law transform as the non-linear input companding to the training audio signal (x).

62. The apparatus (800) according to any one of claims 60 or 61, wherein the apparatus is configured to The transformation is applied to the training audio signal (x), Where sgn() is the sign function; μ is a parameter that defines the compression level.

63. The apparatus (800) according to any one of claims 51 to 62, wherein the one or more stream blocks are configured to convert the training audio signal into the training result signal.

64. The apparatus (800) according to any one of claims 51 to 63, wherein the one or more stream blocks are adapted to transform the training audio signal into the training result signal under the guidance of the distorted version of the training audio signal using the affine processing of sample values ​​of the training audio signal or of sample values ​​of a signal derived from the training audio signal, Wherein processing parameters (e.g. s, t) of the affine processing are determined based on sample values ​​of the distorted version of the training audio signal using the neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820).

65. The apparatus (800) according to any one of claims 51 to 64, wherein the apparatus is configured to perform normalized stream processing to derive the training result signal from the training audio signal.

66. A method (900) for providing a processed audio signal (103, 203, 303, 503, 603, 703) based on an input audio signal (101, 201, 301, 501, 601, 701), wherein the method comprises processing (910) a noise signal (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal using one or more stream blocks to obtain the processed audio signal, wherein the method includes adapting (920) processing performed using the one or more stream blocks based on the input audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the method comprises applying (930) a depthwise separable convolution to a representation of the input audio signal to derive a preprocessed representation of the input audio signal; wherein the neural network receives the preprocessed representation of the input audio signal.

67. A method (1000) for providing a processed audio signal (103, 203, 303, 503, 603, 703) based on an input audio signal (101, 201, 301, 501, 601, 701), wherein the method comprises processing (1010) a noise signal (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal using one or more stream blocks to obtain the processed audio signal, wherein the method includes adapting (1020) processing performed using the one or more stream blocks based on the input audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the one or more flow blocks include at least one dual-coupled flow block (510, 610, 710, 720); wherein the dual coupled stream block applies (1030) a first affine transformation (512, 612, 716, 726) to a first portion (502a, 602a, 702a, 731a) of an input signal to be modified by the dual coupled stream block, and Wherein the dual coupled stream block applies (1030) a second affine transformation (514, 614, 718, 728) to a second portion (502b, 602b, 702b, 731b) of the input signal to be modified by the dual coupled stream block.

68. A method (1100) for providing a processed audio signal (103, 203, 303, 503, 603, 703) based on an input audio signal (101, 201, 301, 501, 601, 701), wherein the method comprises processing (1110) a noise signal (102, 202, 302, 502, 602, 702) or a signal derived from the noise signal using one or more stream blocks to obtain the processed audio signal, wherein the method includes adapting (1120) processing performed using the one or more stream blocks based on the input audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); wherein the method comprises obtaining (1130) a preprocessed representation of the input audio signal using a filter bank (130, 230, 540, 830), the filter bank (130, 230, 540, 830) comprising a time resolution and / or a frequency resolution adapted to a time resolution and / or a frequency resolution of a human auditory system; and wherein the neural network receives the preprocessed representation of the input audio signal.

69. A method (1200) for providing neural network parameters for audio processing, The method comprises processing (1210) a training audio signal or a processed version thereof using one or more stream blocks to obtain a training result signal, wherein the method includes adapting (1220) processing performed using the one or more stream blocks based on a distorted version of the training audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); The method includes determining (1230) neural network parameters of the neural network so that the characteristics of the training result audio signal approximate or include predetermined characteristics; wherein the method comprises applying (1240) a depthwise separable convolution to a representation of the distorted version of the training audio signal to derive a preprocessed representation of the distorted version of the training audio signal; Wherein the neural network receives the preprocessed representation of the distorted version of the training audio signal.

70. A method (1300) for providing neural network parameters for audio processing, The method comprises processing (1310) a training audio signal or a processed version thereof using one or more stream blocks to obtain a training result signal, wherein the method includes adapting (1320) processing performed using the one or more stream blocks based on a distorted version of the training audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); The method includes determining (1330) neural network parameters of the neural network so that the characteristics of the training result audio signal approximate or include predetermined characteristics; wherein the one or more flow blocks include at least one dual-coupled flow block; wherein the dual coupled stream block applies (1340) a first affine transformation (512, 612, 716, 726) to a first portion (502a, 602a, 702a, 731a) of an input signal to be modified by the dual coupled stream block, and Wherein the dual coupled stream block applies (1340) a second affine transformation (514, 614, 718, 728) to a second portion (502b, 602b, 702b, 731b) of the input signal to be modified by the dual coupled stream block.

71. A method (1400) for providing neural network parameters for audio processing, The method comprises processing (1410) the training audio signal or a processed version thereof using one or more stream blocks to obtain a training result signal, wherein the method includes adapting (1420) processing performed using the one or more stream blocks based on a distorted version of the training audio signal and using a neural network (120, 220, 320, 520, 620, 712, 714, 722, 724, 820); The method includes determining (1430) neural network parameters of the neural network so that the characteristics of the training result audio signal approximate or include predetermined characteristics; wherein the method comprises obtaining (1440) a preprocessed representation of the distorted version of the training audio signal using a filter bank (130, 230, 540, 830), the filter bank (130, 230, 540, 830) comprising a time resolution and / or a frequency resolution adapted to a time resolution and / or a frequency resolution of a human auditory system; and Wherein the neural network receives the preprocessed representation of the distorted version of the training audio signal.

72. A computer program having a program code for executing the method according to any one of claims 66 to 71 when run on a computer.

Images