Apparatus for providing a processed audio signal, method for providing a processed audio signal, apparatus for providing neural network parameters, and method for providing neural network parameters - Patents.com

Abstract

The present invention describes an apparatus for providing a processed audio signal based on an input audio signal, the apparatus being configured to process a noise signal or a signal derived from the noise signal using one or more flow blocks to obtain a processed audio signal, and to adapt the processing performed using the one or more flow blocks depending on the input audio signal and using a neural network. The present invention also describes an apparatus for providing neural network parameters for audio processing, the apparatus being configured to process a training audio signal or a processed version thereof using one or more flow blocks to obtain a training result signal, the apparatus being configured to adapt the processing performed using the one or more flow blocks depending on a distorted version of the training audio signal and using a neural network, and the apparatus being configured to determine neural network parameters of the neural network such that characteristics of the training result audio signal approximate or include predetermined characteristics. Methods for providing a processed audio signal and methods for providing neural network parameters for audio processing are also provided. The present invention provides a trade-off between effective modeling of flow-based audio signal processing using neural networks and audio signal enhancement capabilities.

Description

[Technical field]

[0001] An embodiment according to the present invention relates to an apparatus for providing a processed audio signal.

[0002] A further embodiment of the present invention relates to a method for providing a processed audio signal.

[0003] A further embodiment of the present invention relates to an apparatus for providing neural network parameters.

[0004] A further embodiment of the present invention relates to a method for providing neural network parameters.

[0005] TECHNICAL FIELD Embodiments of the present application relate to audio signal processing using neural networks, in particular audio signal enhancement, especially speech enhancement.

[0006] According to one aspect, embodiments of the present invention can be applied to provide direct enhancement of noisy utterances by neural networks. [Background technology]

[0007] Currently, there are many known approaches to audio enhancement, particularly speech enhancement, which involves distinguishing a target speech signal from an intrusive background. The aim of speech enhancement is to enhance a target speech signal from an interfering background in order to ensure better intelligibility of the speech content. Speech enhancement is important for a wide range of applications, including, for example, hearing aids or automatic speech recognition.

[0008] In recent years, different generative approaches for speech enhancement have been increasingly used, such as variational autoencoders, generative adversarial networks (GANs), and autoregressive models.

[0009] In view of the above, it is desired to create a concept for audio signal enhancement that offers an improved trade-off between computational complexity and achievable audio quality.

[0010] This object is achieved by the subject matter of the pending independent claims. Further advantageous aspects are the subject matter of the dependent claims. Summary of the Invention

[0011] An embodiment of the present invention provides a method for generating a processed speech signal, e.g. an enhanced audio signal, e.g. an enhanced speech signal, or e.g. an enhanced general audio signal, e.g. an enhanced speech signal, based on an input audio signal, e.g. a speech signal, e.g. a distorted audio signal, e.g. a noisy speech signal y, e.g. a clean signal x extracted from the noisy speech signal y. The apparatus forms an apparatus for providing a processed audio signal, e.g., a signal having a magnitude of 0.01, e.g., 0.1, where y=x+n, e.g., n is noise, e.g., a noisy background, using, e.g., affine scaling, or a series of affine scaling operations, to provide a processed audio signal, e.g., an enhanced audio signal, e.g., The device is configured to process a noise signal, e.g. z, or a signal derived from the noise signal, e.g. using one or more flow blocks, e.g. eight flow blocks, e.g. including an affine combination layer, e.g. including a reversible convolution, to obtain TIFF0007641371000002.tif73. The device is configured to adapt the processing performed using the neural network, e.g. using one or more flow blocks, in response to an input audio signal, e.g. a distorted audio signal, e.g. in response to a noisy speech signal y, e.g. in response to noisy time-domain speech samples. The neural network provides one or more processing parameters for the flow blocks, e.g. parameters of the affine processing, e.g. scaling coefficients and shift values, based on e.g. the distorted audio signal, preferably also in response to at least a portion of the noise signal or a processed version thereof.

[0012] This embodiment is based on the finding that processing of an audio signal, for example for speech enhancement purposes, can be performed directly using a flow block processing, which may model, for example, a generative process. It has been found that the flow block processing makes it possible to process a noise signal, for example a noise signal z, generated, for example, by a device or stored, for example, in a device, in a manner adjusted to an input audio signal, for example a noisy audio signal y. The noise signal z represents (or includes) a given (for example simple or complex) probability distribution, preferably a Gaussian probability distribution. It has been found that upon processing of the noise signal conditioned on the distorted audio signal, an enhanced clean part of the input audio signal is provided as a result of the processing, without introducing this clean part, for example a clean part without a noisy background, as input to the device.

[0013] The proposed device provides an effective and easily implemented audio signal processing, in particular direct audio signal processing, e.g. direct enhancement of speech samples. At the same time, high performance, e.g. improved speech enhancement or e.g. improved quality of the processed audio signal, is provided in the proposed device.

[0014] In conclusion, the concepts described herein offer an improved compromise between computational complexity and achievable audio quality.

[0015] According to an embodiment, the input audio signal is represented by a set of time-domain audio samples, e.g. noisy time-domain audio, e.g. speech, samples, e.g. time-domain speech utterances. For example, the time-domain audio samples of the input audio signal, or time-domain audio samples derived therefrom, are input to a neural network, and for example, the time-domain audio samples of the input audio signal, or time-domain audio samples derived therefrom, are processed in the neural network in the form of a time-domain representation, without applying a transformation to convert the domain representation, e.g. the spectral domain representation.

[0016] Performing flow block processing directly in the speech domain (or time domain) allows audio signal processing without the need for any predefined features or time-frequency, TF transformation. No upsampling layer in the generative process is required since both the noise signal and the input audio signal are of the same dimension. Furthermore, it is recognized that processing of time domain samples allows efficient modification of signal statistics in a series of flow blocks performing reversible affine processing and also allows derivation of an audio signal from the noise signal in such a series of flow blocks. It is found that processing of time domain samples in a series of flow blocks allows adapting signal characteristics so that the reconstructed audio signal contains a good auditory impression. Furthermore, it is recognized that by performing processing in the time domain, resource-consuming conversion operations between different signal representation domains can be avoided. Furthermore, it is recognized that performing flow block processing directly in the speech domain (or time domain) reduces the amount of parameters of flow block processing using neural networks. Thus, a computationally less intensive audio signal processing is provided.

[0017] According to an embodiment, a neural network associated with a given flow block of the one or more flow blocks, e.g. a given stage of affine processing, is configured to determine one or more processing parameters of the given flow block, e.g. a scaling factor, e.g. S, and e.g. a shift value, e.g. T, in response to a noise signal, e.g. z, or a signal derived from the noise signal, and in response to an input audio signal, e.g. y.

[0018] Determining one or more processing parameters of the affine processing using a neural network that also receives and processes time-domain samples of the input audio signal allows for controlling synthesis of a processed audio signal based on a noise signal in response to the input audio signal. Thus, the neural network can be trained to provide appropriate processing parameters for the affine processing based on the input audio signal (typically also in response to a portion of the noise signal or a portion of the processed noise signal). It is also recognized that training of the neural network is possible with reasonable effort using a training structure that includes an affine processing that is inverse to the affine processing used to derive the processed audio signal.

[0019] According to an embodiment, a neural network associated with a given flow block, e.g. a given stage of the affine processing, is configured to provide one or more parameters of the affine processing, e.g. in the affine connection layer, e.g. a scaling factor, e.g. S, and e.g. a shift value, e.g. T, which are applied to the noise signal, or a processed version of the noise signal, or a part of the noise signal, or a part of the processed version of the noise signal, e.g. z, during processing.

[0020] By using a neural network to provide one or more parameters of the affine processing, and by applying the affine processing to, for example, the noise signal or a processed version of the noise signal, the processing applied to the noise signal can be inverted. Thus, feeding the complete noise signal through the neural network can be avoided, which would normally result in a non-reversible operation. However, by using a neural network to control the reversible (affine) processing, the training of the neural network can be greatly facilitated, which results in the complexity of the processing that can be handled.

[0021] According to an embodiment, the neural network associated with a given flow block, e.g. a given stage of the affine processing, is configured to determine one or more parameters of the affine processing, e.g. a scaling factor, e.g. S, and e.g. a shift value, e.g. T, in response to a first portion of the flow block input signal, e.g. z, e.g. z1, or in response to a first portion of the preprocessed flow block input signal, e.g. z', and in response to the input audio signal, e.g. y. The affine processing associated with the given flow block, e.g. a given stage of the affine processing, applies the determined parameters, e.g. the scaling factor, e.g. S, and e.g. the shift value, e.g. T, to a second portion of the flow block input signal, e.g. z2, or to a second portion of the preprocessed flow block input signal, e.g. z', to generate an affine processed signal. TIFF0007641371000003.tif75. A first portion, e.g. z1, of a flow block input signal that has not been modified by affine processing, e.g. z, or a pre-processed flow block input signal, e.g. z′, and an affine processed signal, e.g. TIFF0007641371000004.tif75 represents the flow block output signal, e.g. z, of a given flow block, e.g. a given stage of affine processing. new , forming, e.g. constructing, a stage output signal. The affine processing of a given flow block, e.g. an affine combination layer, is ensured to generate a processed audio signal by inverting the flow block processing used in training the neural network used.

[0022] According to an embodiment, a neural network associated with a given flow block includes a depthwise separable convolution in an affine process associated with the given flow block. The neural network may include a depthwise separable convolution, for example, instead of any standard convolution conventionally used in neural networks. For example, applying a depthwise separable convolution instead of any other standard convolution may reduce the amount of parameters of the flow block processing using a neural network. For example, applying a depthwise separable convolution of a neural network in combination with performing flow block processing directly in the speech domain (or time domain) may reduce the number of neural network parameters, for example, from 80 million to 20-50 million, for example 25 million. Thus, a computationally less intensive audio signal processing is provided.

[0023] According to an embodiment, the device comprises a processed flow block output signal z' new , the flow block output signal of a given stage of the affine processing, e.g. z new , for example, configured to apply a reversible convolution, for example a 1×1 reversible convolution, to the stage output signal. The reversible convolution may help to ensure that different samples in different flow blocks (or processing stages) are processed by affine processing. Also, the reversible convolution may help to ensure that different samples are fed to neural nets of different (subsequent) flow blocks. Thus, by effectively modifying the statistical properties of a sequence of time-domain samples, the synthesis of a processed audio signal based on a noise signal may be improved.

[0024] According to an embodiment, the device is configured to apply a non-linear compression, e.g. a μ-law transformation, to an input audio signal, e.g. y, before processing the noise signal, e.g. z, in response to the input audio signal, e.g. y. Regarding the advantages of this functionality, reference is made to the following description of the device for providing neural network parameters, in particular the description of the non-linear compression algorithm used in the device for providing neural network parameters.

[0025] According to an embodiment, the apparatus is configured to apply a μ-law transformation, e.g. a μ-law function, to the input audio signal, e.g. y, as a non-linear compression. Regarding the advantages of this functionality, reference is made to the following description of the apparatus for providing neural network parameters, in particular the description of applying a μ-law transformation as a non-linear compression algorithm used in the apparatus for providing neural network parameters.

[0026] According to an embodiment, the device performs the following on an input audio signal: TIFF0007641371000005.tif1348, where for example sgn() is the sign function and μ is a parameter defining the compression level. Regarding the advantages of this function, reference is made to the following description of the device for providing neural network parameters, in particular the description of applying the same transformation as the non-linear compression algorithm used in the device for providing neural network parameters.

[0027] According to an embodiment, the device is configured to apply a nonlinear enhancement, e.g., an inverse μ-law transform, e.g., the inverse of the μ-law transform, to the processed, e.g., enhanced, audio signal. This provides an effective post-processing tool, e.g., an effective post-processing technique for density estimation, to improve the enhancement result and improved performance of the audio signal processing. As a result, an enhanced signal with minimized high frequency addition is provided as an output of the device.

[0028] According to an embodiment, the device may, for example, receive a processed, e.g. enhanced, audio signal (e.g. The method is configured to apply an inverse μ-law transform, e.g., an inverse μ-law function, by inverting the μ-law transform as a nonlinear extension to the TIFF0007641371000006.tif73). Using the inverse μ-law transform provides improved results of modeling the generative process from a noisy input signal to an enhanced output signal, and thus provides improved enhancement performance. This provides an effective post-processing tool, e.g., an effective post-processing technique for density estimation, improving enhancement results and improved performance of audio signal processing.

[0029] According to an embodiment, the device receives a processed, e.g. enhanced, audio signal, e.g. For TIFF0007641371000007.tif73, The method is configured to apply a transform according to TIFF0007641371000008.tif1354, where sgn() is a sign function and μ is a parameter defining an enhancement level. Improved enhancement results and improved performance of audio signal processing are provided. This provides an effective post-processing tool, e.g., an effective post-processing technique for density estimation, to improve the enhancement results and improved performance of audio signal processing.

[0030] According to an embodiment, the neural network parameters of the neural network for processing the noise signal or a signal derived from the noise signal are previously obtained, e.g. stored in the device, e.g. stored in a remote server, using a processing of the training audio signal or a processed version thereof in one or more training flow blocks to obtain a training result signal, the processing of the training audio signal or a processed version thereof using one or more training flow blocks being adapted depending on the distorted version of the training audio signal and the use of the neural network. The neural network parameters of the neural network are determined such that a characteristic, e.g. a probability distribution, of the training result audio signal approximates or includes a predetermined characteristic, e.g. a noise-like characteristic, e.g. a Gaussian distribution. The one or more neural networks used to provide the processed audio signal are identical to the one or more neural networks used to provide the training result signal, and the training flow blocks perform an affine processing that is inverse to the affine processing performed in providing the processed audio signal.

[0031] Thus, an effective training tool for the neural network associated with the flow block is provided, which provides the parameters of the neural network used in the flow block processing in the device. This leads to improved audio signal processing, in particular improved signal enhancement in the device. For example, obtaining the neural network parameters in this way allows efficient training. It is possible to use an inverse processing approach (for example, defined by the inverse affine transformation) in the training and inference of the neural network parameters (deriving the processed audio signal), which leads to high efficiency and well-predictable signal transformation. Thus, a good auditory impression can be achieved with a feasible complexity.

[0032] According to an embodiment, the device is configured to provide neural network parameters of a neural network for processing a noise signal or a signal derived from a noise signal, the device being configured to process the training audio signal or a processed version thereof using one or more flow blocks to obtain a training result signal. The device is configured to adapt the processing of the training audio signal or a processed version thereof, performed using one or more flow blocks, depending on the distorted version of the training audio signal and using the neural network. The device is configured to determine neural network parameters of the neural network, for example using a parameter optimization procedure, for example using an evaluation of a cost function, for example an optimization function, such that a characteristic, for example a probability distribution, of the training result audio signal approximates or includes a predefined characteristic, for example a noise-like characteristic, for example a Gaussian distribution. Using a parameter optimization procedure may for example reduce the number of neural network parameters, for example from 80 million to 20-50 million, for example 25 million. The device is configured to provide neural network parameters for a neural network associated with a flow block used for processing the audio signal in the device. Thus, the device provides an effective training tool for the neural network associated with the flow block without the need for an external training tool.

[0033] According to an embodiment, the apparatus comprises an apparatus for providing neural network parameters, the apparatus for providing neural network parameters being configured to provide neural network parameters of a neural network for processing a noise signal or a signal derived from a noise signal. The apparatus for providing neural network parameters is configured to process a training audio signal or a processed version thereof using one or more training flow blocks to obtain a training result signal. The apparatus for providing neural network parameters is configured to adapt the processing of the training audio signal or a processed version thereof, performed using one or more flow blocks, depending on the distorted version of the training audio signal and using the neural network. The apparatus is configured to determine neural network parameters of the neural network, for example using an evaluation of a cost function, for example an optimization function, for example using a parameter optimization procedure, such that a characteristic, for example a probability distribution, of the training result audio signal approximates or includes a predetermined characteristic, for example a noise-like characteristic, for example a Gaussian distribution. Thus, the apparatus comprises an effective training tool for the neural network associated with the flow blocks without the need for an external training tool.

[0034] According to an embodiment, one or more flow blocks are configured to synthesize a processed audio, e.g., a speech signal, based on a noise signal under the guidance of an input audio, e.g., a speech signal. Thus, the input audio signal may serve as an input quantity of the neural network, thereby controlling the synthesis of the processed audio signal based on the noise signal. For example, the neural network may effectively control an affine process to approximate the signal characteristics of the noise signal (or a processed version thereof) to the (statistical) signal characteristics of the input audio signal, and the noise contribution of the input audio signal is at least partially reduced. Thus, an improvement in the signal quality of the processed audio signal compared to the input audio signal may be achieved.

[0035] According to an embodiment, one or more flow blocks are configured to synthesize a processed audio, e.g. speech, signal based on the noise signal under the guidance of an input audio, e.g. speech, signal using affine processing of sample values ​​of a noise signal or a signal derived from the noise signal. Processing parameters of the affine processing, e.g. a scaling factor, e.g. S, and e.g. a shift value, e.g. T, are determined based on the sample values ​​of the input audio signal, e.g. in the time domain, using a neural network. It has been found that such processing results in good resulting processed audio signal quality at a reasonable processing load.

[0036] According to an embodiment, the device is configured to perform a normalization flow process, for example to derive a processed audio signal from a noise signal under the guidance of an input audio signal. It is recognized that the normalization flow process provides the ability to successfully generate high quality samples of the processed audio signal in audio enhancement applications.

[0037] An embodiment according to the present invention forms a method for providing a processed audio signal based on an input audio signal, the method comprising processing a noise signal or a signal derived from the noise signal using one or more flow blocks to obtain a processed audio signal, the method comprising adapting the processing performed using the one or more flow blocks depending on the input audio signal, e.g. a distorted audio signal, and using a neural network.

[0038] The method according to this embodiment is based on the same considerations as the device for providing a processed audio signal described above. Moreover, this disclosed embodiment may optionally be supplemented by any other features, functions and details disclosed herein in relation to the device for providing a processed audio signal, both individually and in combination.

[0039] An embodiment according to the present invention forms an apparatus for providing neural network parameters, e.g. edge weights, e.g. θ, of a neural network providing scaling coefficients, e.g. s and shift values, e.g. t, based on a portion of a clean audio signal, e.g. x1, or a processed version thereof, and based on a distorted audio signal, e.g. y, in an inference mode, which may correspond to edge weights of the neural network providing scaling coefficients, e.g. s and shift values, e.g. t, based on a portion of a noise signal, e.g. z, or a processed version thereof, and based on an input audio signal, e.g. y, in an inference mode. The apparatus is configured to process a training audio signal, e.g. a speech signal, e.g. x, or a processed version thereof, using one or more flow blocks, e.g. 8 flow blocks, e.g. using a flow block system, e.g. including a reversible convolution, e.g. including an affine connection layer, for example, in multiple iterations to obtain a training result signal that should be equal to the noise signal. The device is configured to adapt the processing performed using one or more flow blocks in response to a distorted version of a training audio signal, e.g. y, e.g. a distorted audio signal, e.g. a noisy speech signal y, and using a neural network. The neural network provides one or more processing parameters of the flow blocks, e.g. parameters of affine processing, e.g. scaling factors and shift values, e.g. based on the distorted version of the training audio signal, preferably also in response to at least a part of the training audio signal or a processed version thereof. The device is configured to determine neural network parameters of the neural network, e.g. using a parameter optimization procedure performed by the neural network, e.g. using an evaluation of a cost function, e.g. an optimization function, such that a characteristic, e.g. a probability distribution, of the training result audio signal approximates or includes a predetermined characteristic, e.g. a noise-like characteristic, e.g. a Gaussian distribution.

[0040] This embodiment is based on the finding that flow block processing can be applied to audio signal processing, and in particular that neural network parameters of a neural network used in audio signal processing can be determined by learning a mapping from a simple probability distribution to a more complex probability distribution based on clean speech samples, e.g. x, conditional on a noisy counterpart, e.g. y, e.g. by learning the probability distribution of clean speech. For example, it has been found that the parameters of a neural network associated with a sequence of flow blocks can be well used in inference, i.e. when obtaining a processed audio signal based on a noisy signal. It has also been found that it is easily possible to design inference flow blocks that correspond to training flow blocks and can be controlled using a neural network that uses the trained neural network parameters.

[0041] The proposed device provides an effective training tool for neural networks associated with flow blocks, providing parameters of the neural networks used in audio signal processing, which results in improved audio signal processing, in particular improved audio signal enhancement, using the neural network with the determined neural network parameters, which provides high performance, e.g. improved speech enhancement, or for example improved quality of the processed audio signal.

[0042] According to an embodiment, the device is configured to evaluate a cost function, e.g. a loss function, depending on the characteristics of the obtained training result signal, e.g. a distribution of the obtained noise signal, e.g. a Gaussian function distribution, and a variance δ2 of the obtained noise signal, and processing parameters, e.g. a scaling factor of the flow block, which may depend on the input signal of the respective flow block. The device is configured to determine neural network parameters to reduce or minimize the cost defined by the cost function. The correlation between the modeled generative process and the generative process represented by the modeling is optimized. Furthermore, the cost function serves to adjust the neural network parameters such that the processing in the sequence of flow blocks controlled by the neural network transforms the training audio signal into a signal with desired statistical characteristics (e.g. into a noise-like signal). The deviation between the desired statistical characteristics and the signal provided by the training flow block can be efficiently represented by the cost function. Thus, the neural network parameters can be trained or optimized such that the processing in the training flow block provides a signal whose statistical characteristics approximate the desired (e.g. noise-like) characteristics. In this training, the cost function may be a simple (efficiently computable) training objective function, thus facilitating adaptation of the neural network parameters. The trained neural network parameters thus trained can then be used in an inference process to synthesize a processed audio signal based on a noise signal.

[0043] According to an embodiment, the training audio signal, e.g. x, and / or the distorted version of the training audio signal, e.g. y, are represented by a set of time-domain audio samples, e.g. noisy time-domain audio, e.g. speech, samples, e.g. time-domain speech utterances. The time-domain audio samples of the input audio signal, or time-domain audio samples derived therefrom, are, for example, input to a neural network. The time-domain audio samples of the training audio signal, or time-domain audio samples derived therefrom, are, for example, processed in the neural network in the form of a time-domain representation, without applying a transformation to convert the domain representation, e.g. a spectral domain representation. Performing flow block processing directly in the speech domain (or time domain) allows audio signal processing without the need for any predefined feature or time-frequency, TF transformation. If both the training audio signal and the distorted version of the training audio signal are of the same dimension, no upsampling layer is required in the processing. Furthermore, reference is also made to the above-mentioned advantages of time-domain processing.

[0044] According to an embodiment, a neural network associated with a given flow block of the one or more flow blocks, e.g. a given stage of the affine processing, is configured to determine one or more processing parameters for the given flow block, e.g. a scaling factor, e.g. s, and e.g. a shift value, e.g. t, in response to a training audio signal, e.g. x, or a signal derived from the training audio signal, and in response to a distorted version of the training audio signal, e.g. y. Regarding the advantages of this function, reference is also made to the above description of the device for providing a processed audio signal.

[0045] According to an embodiment, a neural network associated with a given flow block, e.g. a given stage of the affine processing, is configured to provide one or more parameters of the affine processing, e.g. in the affine combination layer, e.g. a scaling factor, e.g. s, and e.g. a shift value, e.g. t, which are applied to a training audio signal, e.g. x, or to a processed version of the training audio signal, or to a part of the training audio signal, or to a part of the processed version of the training audio signal being processed. Regarding the advantages of this function, reference is also made above to the description of the device for providing a processed audio signal.

[0046] According to an embodiment, a neural network associated with a given flow block, e.g. a given stage of the affine processing, is configured to determine one or more parameters of the affine processing, e.g. a scaling factor, e.g. s, and e.g. a shift value, e.g. t, in response to a first portion of a flow block input signal, e.g. x, e.g. x1, or in response to a first portion of a preprocessed flow block input signal, e.g. x', and in response to a distorted version of a training audio signal, e.g. y. The affine processing associated with the given flow block, e.g. a given stage of the affine processing, applies the determined parameters to a second portion of the flow block input signal x, e.g. x2, or to a second portion of the preprocessed flow block input signal x', to obtain an affine processed signal, e.g. TIFF0007641371000009.tif75, for example. TIFF0007641371000010.tif75 represents the flow block output signal of a given flow block, e.g., a given stage of affine processing, e.g., the stage output signal x new, forming, e.g., constructing, a stage output signal. Affine processing of a given flow block, e.g., an affine combination layer, ensures the reversibility of the flow block processing and the efficient calculation of the Jacobian determinant used in determining the characteristics, e.g., the probability density function, of the training result audio signal. Furthermore, by affine processing only a part of the flow block input signal, while leaving other parts of the flow block input signal unchanged by the affine processing, the reversibility of the processing is achieved while still having the opportunity to input a part of the flow block input signal to the neural network. That part of the flow block input signal used as the input of the neural network is not affected by the affine processing, and thus is available both before and after the affine processing if the affine processing is reversible (which is usually the case), thus allowing the reversal of the processing direction (when going from the training stage to the inference stage). Thus, the learned neural network coefficients learned during training are very significant in the inference stage.

[0047] According to an embodiment, a neural network associated with a given flow block includes a depthwise separable convolution in an affine process associated with the given flow block. The neural network may include a depthwise separable convolution, for example, instead of any standard convolution conventionally used in neural networks. For example, applying a depthwise separable convolution instead of any other standard convolution may reduce the amount of parameters of flow block processing using a neural network. For example, applying a depthwise separable convolution of a neural network in combination with performing flow block processing directly in the speech domain (or time domain) may reduce the number of neural network parameters, for example, from 80 million to 20-50 million, for example 25 million. Thus, a computationally less intensive audio signal processing is provided.

[0048] According to an embodiment, the apparatus is configured to apply an invertible convolution, e.g. 1×1, to a flow block input signal of a given stage of the affine processing, e.g. x, e.g. a stage input signal, which may e.g. be the training audio signal of the first stage or a signal derived from the training audio signal, e.g. an output signal of a previous stage of another subsequent stage following the first stage, in order to obtain a preprocessed flow block input signal, e.g. x', a preprocessed version of the flow block input signal, e.g. a convolved version of the flow block input signal. Regarding the advantages of this feature, reference is made to the above description of the apparatus for providing a processed audio signal.

[0049] According to an embodiment, the device is configured to apply a nonlinear input compression, e.g. a μ-law transformation, to the training audio signal, e.g. x, before processing the training audio signal, e.g. x. A nonlinear compression algorithm is applied to map small amplitudes of the audio data samples to wider amplitudes and larger amplitudes to smaller intervals. This solves the problem that higher absolute amplitudes are not well represented in the clean data samples. This provides an effective pre-processing tool, e.g. an effective pre-processing technique for density estimation, thereby improving the enhanced results and improved performance of audio signal processing using a neural network with determined neural network parameters. Regarding the advantages of this function, reference is also made to the above description of the device for providing a processed audio signal. The nonlinear input compression may, for example, be the inverse of the nonlinear expansion described above.

[0050] According to an embodiment, the device is configured to apply a μ-law transformation, e.g. a μ-law function, as a nonlinear input compression to a training audio signal, e.g. x. Instead of learning the distribution of a clean signal, the device learns the distribution of a compressed signal. Flow processing using the μ-law transformation can capture finer grained speech parts with less background leakage. Thus, improved enhancement performance is provided during audio signal processing using a neural network with determined neural network parameters. With regard to the advantages of this feature, reference is also made to the above description of the device for providing a processed audio signal. The μ-law transformation may, for example, be (at least approximately) the inverse of the transformation described above with respect to the device for providing a processed audio signal. This provides an effective pre-processing tool, e.g. an effective pre-processing technique for density estimation, thereby improving the enhancement results and improved performance of audio signal processing using a neural network with determined neural network parameters.

[0051] According to an embodiment, the device performs the following on a training audio signal (x): TIFF0007641371000011.tif1348, where sgn() is the sign function and μ is a parameter defining the compression level. Improved enhancement results and improved performance of audio signal processing using a neural network with the determined neural network parameters are provided. With regard to the advantages of this feature, reference is also made to the above description of an apparatus for providing a processed audio signal. The transformation may, for example, be (at least approximately) the inverse of the transformation described above with respect to the apparatus for providing a processed audio signal. This provides an effective pre-processing tool, for example an effective pre-processing technique for density estimation, thereby improving the enhancement results and improved performance of audio signal processing using a neural network with the determined neural network parameters.

[0052] According to an embodiment, the device is configured to, in response to a distorted version of a training audio signal, e.g. y, apply a non-linear input compression, e.g. a μ-law transformation, to the distorted version of the training audio signal, e.g. y, before processing the training audio signal, e.g. x. Regarding the advantages of this functionality, reference is made to the above description of the non-linear compression algorithm used to process the training audio signal, e.g. x.

[0053] According to an embodiment, the device is configured to apply a μ-law transform, e.g. a μ-law function, as a non-linear input compression to a distorted version of a training audio signal, e.g. y. Regarding the advantages of this functionality, reference is made to the above description of applying a μ-law transform as a non-linear compression algorithm used to process a training audio signal, e.g. x.

[0054] According to an embodiment, the device performs on the distorted version of the training audio signal: TIFF0007641371000012.tif1348, where sgn() is the sign function and μ is a parameter defining the compression level. Regarding the advantages of this function, reference is made to the above description of applying the same transform as the non-linear compression algorithm used to process the training audio signal, for example x.

[0055] According to an embodiment, one or more flow blocks are configured to convert a training audio signal into a training result signal that approximates a noise signal or includes noise-like characteristics. It has been found that neural networks associated with the flow blocks and trained to convert training audio signals into noise signals (or at least noise-like signals) can be well used for speech enhancement (e.g., using an "inverse" inference flow block to perform a substantially inverse function compared to the function of the training flow block).

[0056] According to an embodiment, one or more flow blocks are adjusted, for example by suitable determination of neural network parameters, to convert the training audio signal into a training result signal under the guidance of a distorted version of the training audio signal, for example a speech signal, using affine processing of sample values ​​of the training audio signal or a signal derived from the training audio signal. Processing parameters of the affine processing, for example a scaling factor, for example s, and for example a shift value, for example t, are determined based on, for example, time-domain sample values ​​of the distorted version of the training audio signal using a neural network. It has been found that the neural network used to adjust one or more flow blocks (for example by providing a scaling value and / or a shift value) can be well used for audio enhancement in an inference device (for example an device for providing a processed audio signal as described herein).

[0057] According to an embodiment, the device is configured to perform a normalized flow process to derive a training result signal from the training audio signal, for example under the guidance of a distorted version of the training audio signal. Normalizing the flow process provides the ability to successfully generate high quality samples of the training result signal. It has also been found that the normalized flow process provides good results for speech enhancement using neural network parameters obtained by training.

[0058] An embodiment according to the present invention forms a method for providing neural network parameters, e.g. edge weights, e.g. θ, of a neural network providing scaling coefficients, e.g. s and shift values, e.g. t, based on a portion of a clean audio signal, e.g. x1, or a processed version thereof, and based on a distorted audio signal, e.g. y, in a training mode, which may correspond to edge weights of the neural network providing scaling coefficients, e.g. s and shift values, e.g. t, based on a portion of a noise signal, e.g. z, or a processed version thereof, and based on an input audio signal, e.g. y, in an inference mode. The method comprises processing a training audio signal, e.g. speech, e.g. x, or a processed version thereof, using one or more flow blocks, e.g. using a flow block system, e.g. including a reversible convolution, e.g. including an affine connection layer, for example, in multiple iterations, to obtain a training result signal, which must be equal to a noise signal, e.g. z. The method comprises adapting the processing performed using one or more flow blocks depending on a distorted version, e.g. y, of the training audio signal, e.g. a distorted audio signal, e.g. a noisy speech signal y, and using the neural network. The neural network provides one or more processing parameters of the flow block, e.g. affine processing parameters such as scaling factors and shift values, e.g. based on a distorted version of the training audio signal, preferably also in response to at least a portion of the training audio signal or the processed version thereof. The method includes determining neural network parameters of the neural network, e.g. using a parameter optimization procedure, e.g. using an evaluation of a cost function, such that a characteristic, e.g. a probability distribution, of the training result audio signal approximates or includes a predetermined characteristic, e.g. a noise-like characteristic, e.g. a Gaussian distribution.

[0059] The method according to this embodiment is based on the same considerations as the device for providing neural network parameters described above. Moreover, this disclosed embodiment may be optionally supplemented by any other features, functions and details disclosed herein in relation to the device for providing neural network parameters, both individually and in combination.

[0060] An embodiment according to the present invention forms a computer program having a program code for performing the method according to any of the above described embodiments, when the computer program is run on a computer.

[0061] The apparatus for providing a processed audio signal, the method for providing a processed audio signal, the apparatus for providing neural network parameters, the method for providing neural network parameters, and the computer programs for implementing these methods may optionally be supplemented by any of the features, functions and details disclosed in this specification (the entire document), both individually and in any combination.

[0062] Preferred embodiments of the present application are described below on the basis of the drawings. [Brief description of the drawings]

[0063] [Figure 1] 1 shows a schematic diagram of an apparatus for providing a processed signal according to an embodiment. [Diagram 2] 1 shows a schematic diagram of an apparatus for providing a processed signal according to an embodiment. [Diagram 3] 1 shows a schematic diagram of an inference flow block of an apparatus for providing a processed signal according to an embodiment; [Figure 4] 1 shows a schematic diagram of an apparatus for providing a processed signal according to an embodiment. [Diagram 5] FIG. 1 shows a schematic diagram of an apparatus for providing neural network parameters according to an embodiment. [Figure 6]FIG. 1 shows a schematic diagram of an apparatus for providing neural network parameters according to an embodiment. [Figure 7] 1 shows a schematic diagram of a training flow block of an apparatus for providing neural network parameters according to an embodiment; [Figure 8] FIG. 1 shows a schematic diagram of an apparatus for providing neural network parameters according to an embodiment. [Figure 9] 1 shows a diagram of providing non-linear input compression (compression and expansion) in an apparatus for providing a processed signal according to an embodiment or an apparatus for providing neural network parameters according to an embodiment; [Figure 10] 1 illustrates a flow block system for audio signal processing according to an embodiment. [Figure 11] 1 shows a table comparing the apparatus and method according to the embodiment with the prior art. [Figure 12] 1 shows a graphical representation illustrating the performance of an apparatus and method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] FIG. 1 shows a schematic diagram of an apparatus 100 for providing a processed audio signal according to an embodiment.

[0065] The apparatus 100 is configured to provide a processed, e.g. enhanced, audio signal 160 based on an input audio signal y, 130. The processing may be carried out, for example, by N flow blocks, e.g. an inference flow block 110, associated with a neural network (not shown). 1...N The flow is executed in flow block 110. 1...N is configured to process an incoming audio signal, for example a speech signal.

[0066] An input audio signal y, 130 is introduced into the device 100 to be processed. The input audio signal y is, for example, a noisy input signal, or for example a distorted audio signal. The input audio signal y, 130 may for example be defined as y=x+n, where x is the clean part of the input signal and n is the noisy background. The input audio signal y, 130 may for example be represented as time-domain audio samples, for example noisy time-domain speech samples.

[0067] The input audio signal y, 130 may optionally be pre-processed, eg compressed, eg by non-linear compression, such as the non-linear compression described with reference to FIG. 9, eg as shown in FIG.

[0068] The input audio signal y and its correspondingly cleaned portion x may optionally be grouped into a vector representation (or a matrix representation).

[0069] A noise signal z, 120 (or a preprocessed version thereof z(i=1)) is introduced into the first flow block 1101 of the apparatus 100 together with an input audio signal y, 130.

[0070] The noise signal z, 120 may be generated, for example, in the apparatus 100, or may be generated, for example, externally and provided to the apparatus 100. The noise signal z, 120 may be stored in the apparatus 100 or provided to the apparatus from an external storage device, for example, a remote server. The noise signal z, 120 may be defined, for example, as sampled from a normal distribution with zero mean and unit variance, for example, z~N(z;0;I). The noise signal z, 120 may be represented, for example, as noise samples, for example, time-domain noise samples.

[0071] The signal z may be pre-processed into a noise signal z(i=1) before entering the device 100 or within the device 100.

[0072] For example, noise samples z of noise signal z, or noise samples of preprocessed noise signal z(i=1), may be arbitrarily grouped into groups of samples, for example groups of 8 samples, for example into a vector representation (or matrix representation).

[0073] The optional pre-processing step is not shown in FIG.

[0074] The noise signal z(i=1), 1401 (or the noise signal z) is introduced into the first flow block 1101, e.g., the inference flow block, of the device 100 together with the input audio signal y, 130. With reference to FIG. 2 and FIG. 3, the first flow block 1101 and the flow block 110 1...N The processing of the noise signal z(i=1), 1401 and the input audio signal y in the subsequent flow blocks of the flow block 110 will be further described. The input signal z(i) is, for example, adjusted by the input audio signal y, 130. 1...N (or, in general, 110 i ) The input audio signal y, 130 is processed, for example, in the flow block 110 1...N is introduced into each flow block.

[0075] After processing in the first flow block 1101 of the noise signal z(i=1), 1401, the output signal z new (i=1), 1501 is output. Signal z new (i=1), 1501 is the input signal z(i=2), 1402 for the second flow block 1102 of the device 100 together with the input audio signal y, 130. The output signal z(i=2), 1402 for the second flow block 1102 of the device 100 new (i=2), 1502 is the input signal z of the third block (i=3), etc. N is the input signal z(i=N), 140 N and forming the output signal 160 of the device 100. new (i=N), 150 N Outputs the signal z new (i=N), 150N is a processed audio signal, e.g., an enhanced audio signal, e.g., representing the enhanced clean portion of the input audio signal y, 130. Form TIFF0007641371000013.tif73, 160.

[0076] The clean portion x of the input audio signal y, 130 is not introduced separately to the apparatus 100. The apparatus 100 processes, e.g., a generated noise signal z, 120 based on the input audio signal y, 130 to receive, e.g., generate, e.g., output, an enhanced audio signal, which is, e.g., an enhancement of the clean portion of the input audio signal y, 130.

[0077] Generally speaking, the apparatus includes one or more flow blocks 1101 to 110 to obtain a processed (e.g., enhanced) audio signal 160. N It can be said that the apparatus 100 is configured to process a noise signal (e.g., noise signal z) or a signal derived from a noise signal (e.g., pre-processed noise signal z(i=1)) using a neural network. Generally speaking, the apparatus 100 processes one or more of the flow blocks 1101 to 110 using a neural network in response to an input audio signal (e.g., distorted audio signal y) and (e.g., based on the distorted audio signal, and preferably also in response to at least a part of the noise signal, one or more processing parameters of the flow blocks, e.g., parameters of affine processing such as scaling factors and shift values, or processed versions thereof, can be provided based on the distorted audio signal. N The method is configured to adapt the processing performed using

[0078] However, it should be noted that the device 100 may be optionally supplemented with any of the features, functions, and details disclosed herein, either individually or in combination.

[0079] FIG. 2 shows a schematic diagram of an apparatus 200 for providing a processed signal according to an embodiment.

[0080] In an embodiment, the features, functions and details of the device 100 shown in FIG. 1 may be optionally (individually and in combination) implemented in the device 200, or vice versa.

[0081] The apparatus 200 generates a processed, e.g. enhanced, audio signal y, 230 based on the input audio signal y. The process is configured to provide N flow blocks, e.g., inference flow block 210, associated with a neural network (not shown). 1...N The process is executed in flow block 210. 1...N is configured to process an incoming audio signal, for example a speech signal.

[0082] An input audio signal y, 230 is introduced into the device 200 to be processed. The input audio signal y is, for example, a noisy input signal, or for example a distorted audio signal. The input audio signal y, 230 is, for example, defined as y=x+n, where x is the clean part of the input signal and n is the noisy background. The input audio signal y, 230 may, for example, be represented as time-domain audio samples, for example noisy time-domain speech samples.

[0083] The input audio signal y, 230 may optionally be pre-processed, eg compressed, eg by non-linear compression, such as the non-linear compression described with reference to FIG. 9, eg as shown in FIG.

[0084] The input audio signal y and its correspondingly cleaned portion x may optionally be grouped into a vector representation (or a matrix representation).

[0085] The noise signal z, 220 (or a pre-processed version thereof z(i=1)) is introduced into the first flow block 2101 of the apparatus 200 together with the input audio signal y, 230. The noise signal z, 220 may be generated, for example, in the apparatus 200, or may be generated, for example, externally and provided to the apparatus 200. The noise signal z may be stored in the apparatus 200 or provided to the apparatus from an external storage, for example a remote server. The noise signal z, 220 may be defined, for example, as sampled from a normal distribution with zero mean and unit variance, for example z~N(z;0;I). The noise signal z, 220 may be represented, for example, as noise samples, for example time-domain noise samples.

[0086] The signal z, 220 may be pre-processed before being introduced into the apparatus 200. For example, the noise samples of the noise signal z, 220 may be optionally grouped into groups of samples, for example into groups of 8 samples, for example into a vector representation (or matrix representation). The optional pre-processing step is not shown in FIG.

[0087] The noise signal z(i), 2401 is introduced into a first flow block 2101, e.g., an inference flow block, of the device 200 together with the input audio signal y, 230. The noise signal z(i), 2401 is, for example, inferred from the flow block 110 based on being adjusted by the input audio signal y, 230. 1...N The input audio signal y, 230 is processed in the flow block 110 1...N is introduced into each flow block.

[0088] The processing in the first flow block 2101 is performed, for example, in two steps, for example in two blocks (or using two functional blocks), for example in two operations: an affine connection layer 2111 and optionally a 1×1 reversible convolution 2121.

[0089] In the affine coupling layer block 2111, the noise signal z(i=1), 2401 is processed based on being adjusted by the input audio signal y, 230, which is introduced into the affine coupling layer block 2111 of the first flow block 2101. 1...N The affine connection layer block 211 of the subsequent flow block 1...N An example of the processing of the noise signal z(i=1), 2401 and the input audio signal y, 230 in is further described with reference to FIG. 3. After processing in the affine connection layer block 2111 of the first flow block 2101, the output signal z new (i=1), 2501 is output.

[0090] In the reversible convolution block 2121, the output signal z new The sample 2501 of (i=1) is mixed to produce the processed flow block output signal z' new (i=1) is received. The reversible convolution block 2121, for example, reverses (or, in general, changes) the order of the channels at the output of the affine combination layer block 2111. The reversible convolution block 2121 may be implemented, for example, using a weight matrix W, for example as a random rotation matrix, or for example as a pseudo-random but deterministic rotation or permutation matrix. The first flow block 2101 receives an output signal z new (i=1) or the processed flow block output signal z' new (i=1) as the output flow block signal 2511, which is correspondingly the input signal z(i=2), 2402 for the second flow block 2102 of the device 200 together with the input audio signal y, 230. new (i=2), 2502 is the input signal z of the third block (i=3), etc. N is the input signal z(i=N), 240 N and forming the output signal 260 of the device 200. new (i=N), 250 NOutputs the signal z new (i=N), 250 N is a processed audio signal, e.g., an enhanced audio signal, e.g., representing the enhanced clean portion of the input audio signal y, 230. Form TIFF0007641371000015.tif73, 260.

[0091] Flowblock 210 1...N The processing in all subsequent flow blocks may be performed in two steps, e.g., in two blocks, e.g., in two operations: an affine connection layer and a 1×1 reversible convolution. These two steps may, e.g., be the same as those described in connection with the first flow block 2101 (e.g., different neural net parameters may be used in the different flow blocks).

[0092] Flowblock 210 1...N Affine connection layer block 211 1...N is associated with (or includes) a corresponding neural network (not shown), which, as described above, is implemented by the flow block 210 1...N The parameters of the network are predetermined during training of the network, for example by the apparatus (or functionality) described with reference to Figures 5 to 8.

[0093] The clean portion x of the input audio signal y, 230 is not introduced separately to the apparatus 200. The apparatus 200 processes, e.g., a generated noise signal z, 220 based on the input audio signal y, 230 to receive, e.g., generate, e.g., output, an enhanced audio signal, which is, e.g., an enhancement of the clean portion of the input audio signal y, 230.

[0094] However, it should be noted that the device 200 may be optionally supplemented with any of the features, functions, and details disclosed herein, either individually or in combination.

[0095] FIG. 3 shows a schematic diagram of a flow block 311, for example an inference flow block, according to an embodiment.

[0096] Flow block 311 may be part of the processing by, for example, apparatus 100 shown in Figure 1 or apparatus 200 shown in Figure 2. The flow block of apparatus 100 shown in Figure 1 may have the same structure as flow block 311 shown in Figure 3, or may include the functionality (and / or structure) of flow block 311 (e.g., with additional functionality). The affine connection layer block of the flow block of apparatus 200 shown in Figure 2 may have the same structure as flow block 311 shown in Figure 3, or may include the functionality (and / or structure) of flow block 311 (e.g., with additional functionality).

[0097] In FIG. 3 and the following description, the flow block index i is partially omitted for simplicity.

[0098] An input signal 340 is introduced into the flow block. The input signal 340 may represent a noise signal (or a processed version thereof) z(i), for example as shown in the embodiment shown in FIG. 1. For example, the input signal 340 may be represented in the form of time-domain samples. The input signal 340 may be optionally grouped into a vector representation (or matrix representation).

[0099] The input signal 340 is split 370 into two portions z1(i) and z2(i), for example randomly, or pseudo-random but deterministically, or in a predetermined manner (e.g., into two subsequent portions).

[0100] The first portion z1(i) (which may, for example, include a subset of the time-domain samples of the input signal 340) is introduced into a neural network 380 (also referred to as NN(i)) associated with a flow block 311 (having a flow block index i). The neural network 380 may, for example, be associated with the flow block 110 of the device 100 shown in FIG. 1...NThe neural network 380 may be, for example, a neural network associated with the flow block 210 of the device 200 shown in FIG. 1...N The parameters of the neural network 380 may be determined in advance, for example, during training of the network, for example, by the apparatus described with reference to Figures 5 to 8.

[0101] The first part z1(i) is introduced into the neural network 380 together with the input audio signal y, 330. The input audio signal y, 330 may be, for example, a noisy input signal or, for example, a distorted audio signal. The input audio signal y, 330 may be, for example, defined as y=x+n, where x is the clean part of the input audio signal y, 330 and n is the noisy background.

[0102] The input audio signal y, 330 may optionally be pre-processed, eg compressed, eg by non-linear compression, such as the non-linear compression described with reference to FIG. 9, eg as shown in FIG.

[0103] The input audio signal y and its correspondingly cleaned portion x may optionally be grouped into a vector representation (or a matrix representation).

[0104] The neural network 380 processes the first portion z1(i) and the input audio signal y, 330, e.g., processes the first portion z1(i) in response to, e.g., as adjusted by, the input audio signal y, 330. The neural network 380 determines processing parameters, e.g., a scaling factor, e.g., S, and a shift value, e.g., T, which are the output (371) of the neural network 380. The determined parameters S, T have, e.g., a vector representation. For example, different scaling and / or shift values ​​may be associated with different samples of the second portion z2(i). The second portion z2(i) of the noise signal z (which may, e.g., include a subset of the time-domain samples of the input signal 340) is processed (372) using the determined parameters S, T. Processed (Affine Processed) Second Portion Second Portion TIFF0007641371000016.tif75(i) is defined by the following formula: TIFF0007641371000017.tif1318(1) In this equation, s may be equal to S (e.g., if only a single scale factor value is provided by the neural net), or s may be an element of a vector S of scale factor values ​​(e.g., if a vector of scale factor values ​​is provided by the neural net). Similarly, t may be equal to T (e.g., if only a single shift value is provided by the neural net), or t may be an element of a vector T of shift values ​​(e.g., if a vector of scale factor values ​​is provided by the neural net, its entries are associated with different sample values ​​of z2(i)).

[0105] for example, The above formula for TIFF0007641371000018.tif75 may be applied element-by-element to individual elements or groups of elements of the second portion z2, however, if only a single value s and a single value t are provided by the neural net, then this single value s and this single value t may be applied in the same way to all elements of the second portion z2.

[0106] the unprocessed first portion z1(i) of the signal z and the processed portion z TIFF0007641371000019.tif76 is combined (373) and processed in flow block 311 as signal z new , 350. This output signal z new is introduced into the next, e.g., result or subsequent, flow block, e.g., the second flow block, e.g., flow block i+1. For i=N, the signal z new , 350 are, for example, the output signals of the corresponding devices. The file is TIFF0007641371000020.tif73.

[0107] However, it should be noted that the flow block 311 may be optionally supplemented with any of the features, functions and details disclosed herein, either individually or in combination.

[0108] Also, a flow block 311 may optionally be used in any of the embodiments disclosed herein.

[0109] FIG. 4 shows a schematic diagram of an apparatus 400 for providing a processed signal according to an embodiment.

[0110] In an embodiment, features, functions and details of the device 100 shown in FIG. 1 or the device 200 shown in FIG. 2 may optionally be implemented in the device 400 (individually and in combination), or vice versa.

[0111] The flow block 311 shown in FIG. 3 can be used, for example, in the device 400 in the embodiment.

[0112] The apparatus 400 is configured to provide a processed, e.g. enhanced, audio signal based on an input audio signal y, 430. The processing is carried out by N flow blocks, e.g. an inference flow block 410, associated with a neural network (not shown). 1...NThe process is executed in flow block 410. 1...N is configured to process an incoming audio signal, for example a speech signal.

[0113] An input audio signal y, 430 is introduced into the device 400 to be processed. The input audio signal y, 430 is, for example, a noisy input signal, or for example a distorted audio signal. The input audio signal y is, for example, defined as y=x+n, where x is the clean part of the input signal and n is the noisy background. The input audio signal y, 430 may, for example, be represented as time-domain audio samples, for example noisy time-domain speech samples.

[0114] The input audio signal y, 430 may optionally be pre-processed, for example compressed by non-linear compression 490 .

[0115] A non-linear compression step 490 is optionally applied to the input audio signal y, 430. Step 490 is optional as shown in Figure 4. The non-linear compression step 490 can for example be applied to compress the input audio signal y, 430. In an embodiment, the non-linear input compression step 490 is as described with reference to Figure 9.

[0116] In an embodiment, the nonlinear compression 490 may be represented, for example, by a μ-law compression or, for example, a μ-law transformation of the input audio signal y, 430. For example, TIFF0007641371000021.tif1351(2) where sgn() is the sign function, μ is a parameter that defines the compression level.

[0117] The parameter μ may for example be set to 255, a common value used in telecommunications. The input audio signal y and its correspondingly cleaned part x may optionally be grouped into a vector representation (or a matrix representation).

[0118] The noise signal z, 420 may for example be an input signal to the apparatus 400 or may be generated by the apparatus 400. Before introducing the noise signal z, 420 into the first flow block 4101 of the apparatus 400, the audio samples of the noise signal z are grouped (for example in a grouping block 405) into groups of samples, for example groups of 8 samples, for example into a vector representation (or matrix representation). The grouping step 405 is an optional step, as shown in FIG.

[0119] The (arbitrarily grouped) noise signals z(i), 4401 are introduced into the first flow block 4101 of the device 400 together with the input audio signal y, 430 or together with a pre-processed, e.g. compressed, input audio signal y'. The noise signal z, 420 is, e.g. generated in (or by) the device 400 or, e.g. generated externally and provided to the device 400. The noise signal z may be stored in the device 400 or provided to the device from an external storage, e.g. a remote server. The noise signal z, 420 is, e.g. defined as sampled from a normal distribution (or Gaussian distribution) with zero mean and unit variance, e.g. z~N(z;0;I). The noise signal z, 420 is, e.g. represented as noise samples, e.g. time-domain noise samples.

[0120] The (arbitrarily grouped) noise signals z(i), 4401 are introduced into the first flow block 4101 of the device 400 together with the input audio signal y, 430. The noise signals z(i) are, for example, modulated by the input audio signal y, 430 in the flow block 410. 1...N The input audio signal y, 430 is processed (e.g., continuously or stepwise) in the flow block 410. 1...N is introduced into each flow block.

[0121] The processing in the first flow block 24101 is performed in two steps, eg in two blocks, eg in two operations: an affine connection layer 4111 and optionally a 1×1 reversible convolution 4121 .

[0122] In the affine coupling layer block 4111, the noise signal z(i=1), 4401 is processed based on being modulated by the input audio signal y, 430, which is introduced, for example, into the affine coupling layer block 4111 of the first flow block 4101. It should be noted that the affine coupling layer block may include, for example, a single affine coupling layer or multiple affine coupling layers. 1...N Subsequent flow block 410 2...N Affine connection layer block 411 2...N The processing of the noise signal z(i=1), 4401 and the input audio signal y, 430 in may be performed as described with reference to FIG. 3. After processing in the affine connection layer block 4111 of the first flow block 4101, the output signal z new (i=1), 4501 is output.

[0123] In the reversible convolution block 4121, the output signal z new (i=1), 4501 samples are mixed (e.g., permuted or subjected to an invertible matrix operation such as a rotation matrix) to produce the processed flow block output signal z' new (i=1). The reversible convolution block 4121, for example, reverses the order of the channels (or samples) at the output of the affine combination layer block 4111. The reversible convolution block 4121 may be implemented, for example, as a random (or pseudo-random but deterministic) rotation matrix or as a random (or pseudo-random but deterministic) permutation matrix, for example, using a weight matrix W.

[0124] The first flow block 4101 receives an output signal z new (i=1) or the processed flow block output signal z'new (i=1) as the output flow block signal 4511, which is correspondingly the input signal z(i=2), 4402 of the second flow block 4102 of the device 400 together with the input audio signal y, 430. new (i=2) or z' new (i=2), 4502 is the input signal z of the third block (i=3), etc. N is the input signal z(i=N), 440 N and forming the output signal 460 of the device 400. new (i=N) or z' new (i=N), 450 N Outputs the signal z new (i=N) or z' new (i=N), 450 N is the processed audio signal TIFF0007641371000022.tif73, 460, for example, forms an enhanced audio signal representing the enhanced clean portion of the input audio signal y, 430. In an embodiment, the processed audio signal TIFF0007641371000023.tif73, 460 is, for example, the output signal of device 400.

[0125] Flowblock 410 1...N The processing in all subsequent flow blocks of is performed, for example, in two steps, for example, in two blocks, for example, in two operations: an affine connection layer and a 1×1 reversible convolution. These two steps are, for example, the same (for example, qualitatively) as described with respect to the first flow block 4101. However, different neural network coefficients of the neural network for determining the scaling and shifting values ​​may be used in different processing stages. Furthermore, the reversible convolution may also be different in different stages (but may be equal in different stages).

[0126] Flowblock 410 1...N The affine connection layer block of is associated with a corresponding neural network (not shown), which is implemented as a flow block 410 as shown. 1...N is associated with.

[0127] A non-linear extension step 415 optionally extends the processed audio signal TIFF0007641371000024.tif73, 460. Step 415 is optional as shown in FIG. 4. The nonlinear extension step 415 can be, for example, TIFF0007641371000025.tif73, 460 can be applied to extend the processed audio signal, e.g. TIFF0007641371000026.tif73, 460 can be represented by the inverse μ-law transform. For example, TIFF0007641371000027.tif1354(3) where sgn() is the sign function, μ is a parameter that defines the expansion level.

[0128] The parameter μ may be set, for example, to 255, a common value used in telecommunications. 1...N In the case where nonlinear compression is used as a pre-processing step during the training of the neural network associated with

[0129] It should be noted that the clean portion x of the input audio signal y, 430 is not introduced separately to the apparatus 400. The apparatus 400 processes, e.g., a generated noise signal z, 420 based on the input audio signal y, 430 to receive, e.g., generate, e.g., output, an enhanced audio signal, which is, e.g., an enhancement of the clean portion of the input audio signal y, 430.

[0130] However, it should be noted that the device 400 may be optionally supplemented with any of the features, functions, and details disclosed herein, either individually or in combination.

[0131] FIG. 5 shows a schematic diagram of an apparatus 500 for providing neural network parameters according to an embodiment.

[0132] The apparatus 500 may generate neural network parameters (e.g., quantized values ​​for use by the neural network 380, see flow block 110) based on a training audio signal x, 505, e.g., a clean audio signal, and a distorted version of the training audio signal y, 530, e.g., a distorted audio signal. 1...N , 210 1...N , 410 1...N The process may be configured to provide a neural network 580 (NN(i)) associated with 1...N N flow blocks associated with the 1...N The training flow block 510 1...N is for example configured to process an incoming audio signal, for example a speech signal.

[0133] A distorted version of a training audio signal y, 530 is introduced to the apparatus 500 to be processed (or generated by the apparatus 500). The distorted audio signal y is, for example, a noisy input signal. The distorted training audio signal y, 530 is, for example, defined as y=x+n, where x is a clean part of an input signal, for example, the training input signal x, 505, and n is a noisy background. The distorted training audio signal y, 530 may, for example, be represented as a time-domain audio sample, for example a noisy time-domain speech sample.

[0134] The training audio signal x and the corresponding distorted versions of the training audio signal y may optionally be grouped into a vector representation (or a matrix representation).

[0135] The apparatus 500 includes a training flow block 510, which is a training result audio signal 520, e.g. a noise signal, to be mapped to a distribution, e.g. a Gaussian distribution. 1...N Based on the clean-noise (xy) pair according to 1...N (e.g., neural network 380 may correspond to neural network 380, NN(i), or may be equal to each of neural networks 380m NN(i).

[0136] A training audio signal x, 505 is introduced into a first flow block 5101 of the apparatus 500 together with a distorted training audio signal y, 530. The training audio signal x, 505 is, for example, represented as audio samples, for example as time domain samples.

[0137] The training audio signal x may (optionally) be pre-processed into a training audio signal x(i=1) before entering the apparatus 500. For example, the audio samples x of the noise signal x may be grouped into groups of samples, for example groups of 8 samples, for example into a vector representation (or matrix representation). The optional pre-processing step is not shown in FIG.

[0138] A training audio signal x(i=1), 5401, together with a distorted training audio signal y, 530, is introduced into a first flow block 5101, e.g., a training flow block, of the device 500. With reference to Figs. 6 and 7, the flow block 510 1...N The processing of the training audio signal x(i=1), 5401 and the distorted training audio signal y, 530 in the first flow block 5101 and the subsequent flow blocks of the flow block 5101 will be further described. The training audio signal x(i=1), 5401, for example, is adjusted by the distorted training audio signal y, 530 in the flow block 5101. 1...N The distorted training audio signal y, 530 is processed (e.g., continuously or stepwise) in1...N is introduced into each flow block.

[0139] After processing in the first flow block 5101 of the training audio signal x(i=1), 5401, the output signal x new (i=1), 5501 is output. Signal x new (i=1), 5501 is the input signal x(i=2), 5402 to the second flow block 5102 of the apparatus 500 together with the distorted training audio signal y, 530. The output signal x(i=2), 5402 to the second flow block 5102 of the apparatus 500 is new (i=2), 5502 is the input signal x(i=3) of the third flow block, etc. N is the input signal x(i=N), 540 N and signal x new (i=N), 550 N which forms the output signal 520 of the device 500 or a training result audio signal z, 520, e.g. a noise signal (or at least a noise-like signal having similar statistical properties to the noise signal). The training result audio signal z, 520 may optionally be grouped into a vector representation (or a matrix representation).

[0140] Flowblock 510 1...N The processing of the training audio signal x in response to the distorted training audio signal y in 530 is for example carried out iteratively.

[0141] An estimation (or evaluation or assessment) of the training result audio signal z, 520 may be performed, for example after each iteration, to determine or estimate whether a characteristic, for example a distribution (e.g., a distribution of signal values), of the training result audio signal z, 520 approximates a predetermined characteristic, for example a Gaussian distribution. If the characteristic of the training result audio signal z, 520 does not approach the predetermined characteristic (e.g., within a desired tolerance), the neural network parameters may be altered before the subsequent iteration.

[0142] Therefore, the neural network 580 1...NThe neural network parameters of the neural network 580 1...N A series of flow blocks 510 under the control of 1...N A training result audio signal obtained based on processing of the training audio signal in may be determined (e.g., iteratively) to include (or approximate) desired statistical characteristics (e.g., a desired distribution of values) within a (e.g., predetermined) tolerance.

[0143] Neural Network 580 1...N The neural network parameters may be determined, e.g., using a parameter optimization procedure, e.g., using evaluation of a cost function, e.g., an optimization function, such that the characteristics, e.g., probability distribution, of the training result audio signal approximate or include a predetermined characteristic, e.g., a noise-like characteristic, e.g., a Gaussian distribution.

[0144] In the apparatus 500, a clean signal x is input to a training flow block 510. 1...N 580 neural networks associated with 1...N In order to train the neural network 580, the neural network 580 is introduced together with a corresponding distorted, e.g. noisy, audio signal y. Given the training result audio signal 520, the device 500 generates, as a result of the training, a neural network 580. 1...N The neural network parameters, e.g., edge weights (θ), are determined (590).

[0145] The neural network parameters determined by the apparatus 500 can be used, for example, by a neural network associated with a flow block of the apparatus shown in FIGS. 1, 2 and 4 (where the flow block of the apparatus of FIGS. 1, 2 and 4 is, for example, the flow block 510). 1...N (Note that the affine transformation may be configured to perform an affine transformation that is substantially the inverse of that performed by

[0146] However, it should be noted that the device 500 may be optionally supplemented with any of the features, functions, and details disclosed herein, either individually or in combination.

[0147] FIG. 6 shows a schematic diagram of an apparatus 600 for providing neural network parameters according to an embodiment.

[0148] In embodiments, the features, functions and details of the apparatus 600 may optionally be implemented (individually and in combination) in the apparatus 500 shown in FIG. 5, or vice versa.

[0149] The apparatus 600 receives a training audio signal x, 605, e.g., a clean audio signal, and training audio signals y, y input , 630, e.g., based on a distorted audio signal. The process may be implemented by using neural network 580 of FIG. 1...N A neural network (not shown) such as 1...N The process is executed in flow block 610. 1...N is configured to process an incoming audio signal, for example a speech signal.

[0150] A distorted version of a training audio signal y, 630 is introduced into the device 600 to be processed. The distorted audio signal y, 630 is, for example, a noisy input signal. The distorted training audio signal y, 630 is, for example, defined as y=x+n, where x is a clean part of an input signal, for example the training input signal x, 605, and n is the noisy background. The distorted training audio signal y, 630 is, for example, represented as time-domain audio samples, for example noisy time-domain speech samples.

[0151] The training audio signal x and the corresponding distorted versions of the training audio signal y may optionally be grouped into a vector representation (or a matrix representation).

[0152] The apparatus 600 includes a training flow block 610, which is a training result audio signal 620, e.g. a noise signal, to be mapped to a distribution, e.g. a Gaussian distribution. 1...N The apparatus is configured to provide neural network parameters for a neural network (not shown) based on clean-noise (xy) pairs according to

[0153] A training audio signal x, 605 is introduced into a first flow block 6101 of the apparatus 600 together with a distorted training audio signal y, 630. The training audio signal x, 605 may, for example, be represented as audio samples, for example as time domain samples.

[0154] The training audio signal x may be optionally obtained by subtracting the input audio signal x from the input audio signal x before entering or within the device 600. input (i=1), are pre-processed in 606. As shown in Fig. 6, audio samples x of a training audio signal x, having e.g. 16000 samples, are grouped, e.g. into groups of samples, e.g. into 2000 groups of 8 samples, e.g. into a vector representation (or matrix representation).

[0155] Input audio signal x input (i=1), 640 represents the distorted training audio signals y, y input , 630 are introduced into a first flow block 6101 of the device 600, e.g., a training flow block. input (i=1), 640 represents, for example, distorted training audio signals y, y input , 630 based on which the flow block 610 1...N The distorted training audio signals y, y are processed (e.g., continuously or piecemeal) in input , 630 is a flow block 610 1...Nis introduced into each flow block.

[0156] The processing in the first flow block 6101 is performed in two steps, eg in two blocks, eg in two operations: a 1×1 reversible convolution 6121 and an affine connection layer 6111 .

[0157] In the lossless convolution block 6121, the input audio signal x input The samples 640 for (i=1) are mixed (e.g., reordered or subjected to an invertible matrix operation such as a rotation matrix) before being introduced into the affine combination layer block 6111. The reversible convolution block 6121, for example, reverses the order of the channels at the input of the affine combination layer block 6111. The reversible convolution block 6121 may be implemented, for example, as a random rotation matrix, a pseudo-random but deterministic rotation matrix or a permutation matrix, for example, using a weight matrix W. The input audio signal x input (i=1), 640 is processed in a lossless convolution block 6121 to obtain a preprocessed, e.g., convolved, input audio signal x′ input (i=1), 641. For example, the distorted training audio signals y, y input , 630 are not introduced into the reversible convolution block 6121, but serve only as input to the affine combination layer block 6111. The reversible convolution block is optional and may not be present in an embodiment.

[0158] In the affine connection layer block 6111, the preprocessed input audio signal x′ input (i=1), 641 represents, for example, the distorted training audio signals y, y introduced into the affine combination layer block 6111 of the first flow block 6101. input , 630. 1...N The preprocessed input audio signal x' in the affine connection layer block of the subsequent flow block input(i=1), 641 and the distorted training audio signals y, y input , 630 is further described with reference to FIG.

[0159] Flowblock 610 1...N The processing in all subsequent flow blocks of is, for example, performed in two steps, for example in two blocks, for example in two operations: a 1×1 invertible convolution and an affine connection layer. These two steps are, for example, the same (for example, qualitatively) as described with respect to the first flow block 6101 (the neural networks of different processing stages or flow blocks may include different parameters, and the reversible convolutions may be different in different flow blocks or stages).

[0160] Flowblock 610 1...N The affine connection layer blocks are associated with corresponding neural networks (not shown).

[0161] After processing in the affine connection layer block 6111 of the first flow block 6101, the output signal x new (i=1), 6501 is output. Signal x new (i=1), 6501 is the distorted training audio signal y, y input , 630, together with the input signal x input (i=2), 6402. The output signal x new (i=2), 6502 is the input signal x(i=3) of the third block, etc. N is the input signal x input (i=N), 640 N and forming the output signal 620 of the device 600. new (i=N), 650 N Outputs the signal x new (i=N), 650 Nform a training result audio signal z, 620, e.g. a noise signal. The training result audio signal z, 620 may optionally be grouped into a vector representation (or a matrix representation).

[0162] Flowblock 610 1...N The processing of the training audio signal x in response to the distorted training audio signal y, 630 in may be performed, for example iteratively. An estimation (or evaluation or assessment) of the training result audio signal z, 620 may be performed, for example after each iteration, to estimate whether a characteristic, for example a distribution (e.g. a distribution of signal values), of the training result audio signal z, 620 approximates a predetermined characteristic, for example a Gaussian distribution (e.g. within a desired tolerance). If the characteristic of the training result signal z, 620 does not approach the predetermined characteristic, the neural network parameters may be modified before the subsequent iteration.

[0163] Thus, a neural network (e.g., neural network 580 1...N The neural network parameters of the neural network 580 may correspond to 1...N A series of flow blocks 610 under the control of 1...N training result audio signals 620, 650 obtained based on processing of the training audio signals in N may be determined (eg, iteratively) to have (or approximate) desired statistical characteristics (eg, a desired distribution of values) within a (eg, predetermined) tolerance.

[0164] The neural network parameters of the neural network may be determined, e.g. using a parameter optimization procedure, e.g. using evaluation of a cost function, e.g. an optimization function, such that the characteristics, e.g. a probability distribution, of the training result audio signal approximate or include a predetermined characteristic, e.g. a noise-like characteristic, e.g. a Gaussian distribution.

[0165] In the apparatus 600, a clean signal x is input to a training flow block 610. 1...N, together with a corresponding distorted, e.g., noisy, audio signal y to train a neural network (not shown) associated with the audio signal y. Given (or evaluating) the training result audio signal 620, the apparatus 600 determines neural network parameters, e.g., edge weights (θ), of the neural network as a result of the training.

[0166] The neural network parameters determined by the apparatus 600 may be used, for example, by a neural network associated with the flow blocks of the apparatus shown in Figures 1, 2, and 4, for example, in an inference process following training.

[0167] However, it should be noted that the device 600 may be optionally supplemented with any of the features, functions, and details disclosed herein, either individually or in combination.

[0168] FIG. 7 shows a schematic diagram of a flow block 711, eg, a training flow block, according to an embodiment.

[0169] The flow block may be, for example, part of a process by the apparatus 500 shown in Figure 5 or the apparatus 600 shown in Figure 5. The flow block of the apparatus 500 shown in Figure 5 may have, for example, the same structure or functionality as the flow block 711 shown in Figure 7. The affine connection layer block of the flow block of the apparatus 600 shown in Figure 6 may have, for example, the same structure or functionality as the flow block 711 shown in Figure 7.

[0170] 3, or may perform affine processing that is (at least substantially) the inverse of the affine processing performed by flow block 311. As an example, the addition of a shift value t in training flow block 711 may be the inverse of the subtraction of a shift value in inference flow block 311. Similarly, the multiplication with a scaling value s in training flow block 711 may be the inverse of the division by the scaling value s in inference flow block 311. However, the neural network in training flow block 711 may be identical to the neural network in the corresponding inference flow block 311, for example.

[0171] In FIG. 7 and the following description, the flow block index i is partially omitted for simplicity.

[0172] An input signal 740 is introduced into flow block 711. The input signal 740 may be a training audio signal x(i), or a processed version of the training audio signal, e.g. output by a preceding flow block, or a pre-processed, e.g. convolved, input audio signal x' input (i=1).

[0173] An input signal 740 is split 770, for example randomly or pseudo-randomly (but deterministically), into two portions x1(i) and x2(i).

[0174] The first portion x1(i) is introduced into a neural network 780 associated with the flow block 711. The neural network 780 may be, for example, the neural network associated with the flow block 510 of the device 500 shown in FIG. 1...N The neural network 780 may be, for example, a neural network associated with any (or a given) of the flow block 610 of the device 600 shown in FIG. 1...Naffinely connected layer blocks of the neural network associated with the given one.

[0175] The first portion x1(i) is introduced into the neural network 780 together with a distorted training audio signal y,730, which may be, for example, a noisy signal or, for example, a distorted audio signal. The distorted training audio signal y,730 may be, for example, defined as y=x+n, where x is a clean training audio signal, for example, an input signal 740, for example, a clean portion of the distorted training audio signal y,730, and n is the noisy background.

[0176] The training audio signal x and the corresponding distorted versions of the training audio signal y may optionally be grouped into a vector representation (or a matrix representation).

[0177] The neural network 780 processes a first portion x1(i) of the input signal 740 and the distorted training audio signal y,730, e.g., processes the first portion x1(i) adjusted, e.g., in response to the distorted training audio signal y,730. The neural network 780 determines processing parameters, e.g., a scaling factor, e.g., S, and a shift value, e.g., T, which are the output (771) of the neural network 780. The determined parameters S, T have, e.g., a vector representation. The second portion x2(i) of the input signal 740 is processed (772) using the determined parameters S, T.

[0178] Processed second part Second part TIFF0007641371000028.tif732(i) is defined by the following formula: TIFF0007641371000029.tif728(4) In this equation, s may be equal to S (e.g., if only a single scale factor value is provided by the neural net), or s may be an element of a vector S of scale factor values ​​(e.g., if a vector of scale factor values ​​is provided by the neural net). Similarly, t may be equal to T (e.g., if only a single shift value is provided by the neural net), or t may be an element of a vector T of shift values ​​(e.g., if a vector of scale factor values ​​is provided by the neural net, its entries are associated with different sample values ​​of x2(i)).

[0179] for example, The above formula for TIFF0007641371000030.tif75 may be applied element-by-element to individual elements or groups of elements of the second portion x2, however, if only a single value s and a single value t are provided by the neural net, then this single value s and this single value t may be applied in the same way to all elements of the second portion x2.

[0180] an unprocessed first portion x1(i) of signal x and a processed portion x TIFF0007641371000031.tif76 is combined (773) and processed in flow block 711 as signal x new , 750. This output signal x new is introduced into the next, e.g., subsequent, flow block, e.g., the second flow block, e.g., flow block (i+1). For i=N, the signal x new , 750 are the output signals of the corresponding devices, for example z. The output signals z may optionally be grouped into a vector representation (or matrix representation).

[0181] If a pre-processed noise signal x'(i) is used as the input signal 740, the input signal 740 is, for example, pre-mixed to avoid processing of the same x(i) in the flow block 711. For example, pre-processing (e.g. using reversible convolution) can have the effect that different samples (e.g. originating from different original sample positions) (of the training audio signal) are affinely processed in different flow blocks (i.e. to avoid the same subset of samples being affinely processed in each flow block) and different samples (e.g. originating from different original sample positions) (of the training audio signal) serve as input signals of neural networks associated with different flow blocks or processing stages (i.e. to avoid the same subset of samples being input to the neural network in each flow block). However, it is noted that the flow block 711 may be optionally supplemented by any of the features, functions and details disclosed herein, individually or in combination.

[0182] FIG. 8 shows a schematic diagram of an apparatus 800 for providing neural network parameters according to an embodiment.

[0183] In an embodiment, the apparatus 800 may be combined with, for example, the apparatus 500 shown in Figure 5, or may be combined with, for example, the apparatus 600 shown in Figure 8. Also, features, functions and details of the apparatus 800 may optionally be introduced in the apparatus 500 or the apparatus 600 (individually and in combination), or vice versa.

[0184] The flow block 711 shown in FIG. 7 can be used, for example, in the device 800 in the embodiment.

[0185] The apparatus 800 is configured to provide neural network parameters based on a training audio signal x, 805, e.g., a clean audio signal, and a distorted version of the training audio signal y, 830, e.g., a distorted audio signal. The process may, for example, include N flow blocks associated with the neural network, e.g., training flow block 810. 1...N (Only the neural network 8801 of the first flow block 8101 is shown). 1...N is configured to process an incoming audio signal, for example a speech signal.

[0186] A distorted version of a training audio signal y, 830 is introduced into the device 800 to be processed. The distorted audio signal y, 830 is, for example, a noisy input signal. The distorted training audio signal y, 830 is, for example, defined as y=x+n, where x is a clean part of the input signal, for example the training audio signal x, 805, and n is the noisy background. The distorted training audio signal y, 830 is, for example, represented as time-domain audio samples, for example noisy time-domain speech samples.

[0187] The training audio signal x and the corresponding distorted versions of the training audio signal y may optionally be grouped into a vector representation (or a matrix representation).

[0188] A training audio signal x, 805 is introduced into flow block 8101 of the apparatus 800 together with a distorted training audio signal y, 830. The training audio signal x, 805 is, for example, represented as audio samples, for example as time domain samples.

[0189] The apparatus 800 includes a training flow block 810 to be mapped to a distribution, e.g. a Gaussian distribution, of a training result signal 820, e.g. a noise signal. 1...NThe present invention is configured to provide neural network parameters for a neural network (not shown) based on clean-noise (xy) pairs according to

[0190] A nonlinear input compression step 815 is optionally applied to the training audio signal x, 805. Step 815 is optional as shown in FIG. 8. The nonlinear input compression step 815 can be applied, for example, to compress the training audio signal x, 805. 1...N In training the neural network associated with , rather than learning the distribution of a clear utterance, e.g. a clear audio signal x, the distribution of the compressed signal is learned in the presence of any nonlinear input compression step 815. In an embodiment, the nonlinear input compression step 815 is as described with reference to FIG.

[0191] In an embodiment, the nonlinear input compression 815 may be represented, for example, by a μ-law compression of the training audio signal x, 805, or, for example, by a μ-law transformation. For example, TIFF0007641371000032.tif1351(5) where sgn() is the sign function, μ is a parameter that defines the compression level.

[0192] The parameter μ may for example be set to 255, which is a common value used in telecommunications. A non-linear input compression step 815 can be applied if, for example, one wants to ensure that all values ​​whose distribution of the noise signal z is to be learned are uniformly distributed.

[0193] Before introducing the training audio signal x, 805 into the first flow block 8101 of the apparatus 800, the audio samples of the training audio signal x, 805 or the compressed audio samples of the training input signal x' are optionally grouped 816 into groups of samples, for example groups of 8 samples, for example into a vector representation (or matrix representation). The grouping step 816 is an optional step, as shown in FIG.

[0194] A (arbitrarily grouped) training audio signal x(i=1) 8401 is introduced into the flow block 8101 of the device 800 together with a distorted training audio signal y, 830 .

[0195] A non-linear input compression step 815 is optionally also applied to the distorted training audio signal y, 830. Step 815 is optional as shown in Fig. 8. The non-linear input compression step 815 can for example be applied to compress the distorted training audio signal y, 830. In an embodiment, the non-linear input compression step 815 is as described with reference to Fig. 9.

[0196] In an embodiment, the nonlinear input compression 815 may be represented, for example, by a μ-law compression, or μ-law transformation, of the distorted training audio signal y, 830. For example, TIFF0007641371000033.tif1351(6) where sgn() is the sign function, μ is a parameter that defines the compression level.

[0197] The parameter μ may be set, for example, to 255, a common value used in telecommunications.

[0198] A training audio signal x(i=1) 8401 is introduced into a first flow block 8101, e.g. a training flow block, of the device 800 together with a distorted training audio signal y, 830 or together with a pre-processed, e.g. compressed, distorted training audio signal y′. The training audio signal x(i=1) 8401 is, for example, adjusted by the distorted training audio signal y, 830, based on which the training audio signal x(i=1) 8401 is adjusted into a first flow block 8101, e.g. a training flow block, of the device 800 together with a distorted training audio signal y, 830. 1...N The distorted training audio signal y, 830 is processed in flow block 810. 1...N is introduced into each flow block.

[0199] The processing in the first flow block 8101 is performed, for example, in two steps, for example in two blocks, for example in two operations: a 1×1 reversible convolution 8121 and an affine connection layer 8111 .

[0200] In the reversible convolution block 8121, the samples x(i=1) 8401 of the training audio signal are mixed (e.g., permuted or subjected to an invertible matrix operation such as a rotation matrix) before being introduced into the affine combination layer block 8111. The reversible convolution block 8121, for example, reverses (or changes) the order of the channels at the input of the affine combination layer block 8111. The reversible convolution block 8121 may be implemented, for example, as a random rotation matrix or a random but deterministic rotation matrix or permutation matrix, for example, using a weight matrix W. For example, the training audio signal x(i=1) 8401 is processed in the reversible convolution block 8121 to output a pre-processed, e.g., convolved, training audio signal x′(i=1) 8411. For example, the distorted training audio signal y, 830 is not introduced into the reversible convolution block 8121, but serves only as an input to the affine combination layer block 8111. The reversible convolution block is optional and may not be present in some embodiments.

[0201] In the affine coupling layer block 8111, the pre-processed training audio signal x'(i=1) 8411 is processed based on being adjusted by, for example, a distorted training audio signal y, 830, which is introduced in the affine coupling layer block 8111 of the first flow block 8101. 1...N The processing of the preprocessed training audio signal x'(i=1) 8411 and the distorted training audio signal y, 830 in the affine connection layer block of the subsequent flow block of is now described with reference to, for example, FIG.

[0202] Flowblock 810 1...N The processing in all subsequent flow blocks of is, for example, performed in two steps, for example in two blocks, for example in two operations: a 1×1 invertible convolution and an affine connection layer. These two steps are, for example, the same (for example, qualitatively) as described with respect to the first flow block 8101 (the neural networks of different processing stages or flow blocks may include different parameters, and the reversible convolutions may be different in different flow blocks or stages).

[0203] Flowblock 810 1...N The affine connection layer blocks are associated with corresponding networks (only the neural network 8801 of the first flow block 8101 is shown).

[0204] After processing in the affine connection layer block 8111 of the first flow block 8101, the output signal x new (i=1), 8501 is output. Signal 8 new (i=1), 8501 is the input signal x(i=2), 8402 to the second flow block 8102 of the apparatus 800 together with the distorted training audio signal y, 830. The output signal x(i=2), 8402 to the second flow block 8102 of the apparatus 800 is new(i=2), 8502 is the input signal x(i=3) of the third block, etc. N is the input signal x(i=N), 840 N and a signal x forming an output signal 820 of the device 800, e.g. a training result audio signal. new (i=N), 850 N Outputs the signal x new (i=N), 850 N form a training result audio signal z, 820, e.g. a noise signal. The training result audio signal z, 820 may optionally be grouped into a vector representation (or a matrix representation).

[0205] Flowblock 810 1...N The processing of the training audio signal x in response to the distorted training audio signal y, 830 in may be performed, for example iteratively. An estimation (or evaluation or assessment) of the training result signal z, 820 may be performed, for example after each iteration, to estimate whether the characteristics, for example the distribution (e.g. the distribution of signal values), of the training result signal z, 820 approximates a predetermined characteristic, for example a Gaussian distribution (e.g. within a desired tolerance). If the characteristics of the training result audio signal z, 820 do not approach the predetermined characteristic, the neural network parameters may be modified before the subsequent iteration.

[0206] Thus, a neural network (e.g., neural network 580 1...N , 780 1...N The neural network parameters of the neural network 880 may correspond to 1...N A series of flow blocks 810 under the control of 1...N training result audio signals 820, 850 obtained based on processing of the training audio signals in N may be determined (eg, iteratively) to have (or approximate) desired statistical characteristics (eg, a desired distribution of values) within a (eg, predetermined) tolerance.

[0207] The neural network parameters of the neural network may be determined, e.g. using a parameter optimization procedure, e.g. using evaluation of a cost function, e.g. an optimization function, such that the characteristics, e.g. a probability distribution, of the training result audio signal approximate or include a predetermined characteristic, e.g. a noise-like characteristic, e.g. a Gaussian distribution.

[0208] In the apparatus 800, a clean signal x is input to a training flow block 810. 1...N , is introduced along with a corresponding distorted, e.g., noisy, audio signal y to train a neural network associated with y. Considering (or evaluating) the training result signal 820, neural network parameters, e.g., edge weights (θ), of the neural network are determined as a result of the training.

[0209] The neural network parameters determined by the apparatus 800 may be used, for example, by a neural network associated with the flow blocks of the apparatus shown in Figures 1, 2, and 4, for example, in an inference process following training.

[0210] However, it should be noted that the device 800 may be optionally supplemented with any of the features, functions, and details disclosed herein, either individually or in combination.

[0211] In the following, some considerations underlying embodiments of the present invention are described, e.g., a formulation of the problem is provided, the basics of normalization flow are described, and speech enhancement flow is explained. The concepts described below can be used individually and in combination with the embodiments described herein.

[0212] The flow block processing used in devices 100, 200, 400 shown in Figures 1, 2 and 4, and in devices 500, 600 and 800 shown in Figures 5, 6 and 8, can be described as a conversion from a simple probability distribution to a more complex probability distribution using a reversible and differentiable mapping, for example, formally expressed as follows:

[0213] x = f(z) (7) Where: TIFF0007641371000034.tif714 and TIFF0007641371000035.tif713 is a D-dimensional random variable, TIFF0007641371000036.tif73 is a functional mapping from z to x.

[0214] (5) represents a differentiable and reversible transformation by a differentiable inverse function. The invertibility of TIFF0007641371000037.tif73 ensures that this step can be reversed from x back to z: TIFF0007641371000038.tif722(8) In addition, the function If TIFF0007641371000039.tif73 is invertible and differentiable, then it is guaranteed that the construction of a sequence of 1 to T transformations is also invertible and can be described by a neural network: TIFF0007641371000040.tif719... TIFF0007641371000041.tif712(9) This is followed by the log probability density function, e.g. the log likelihood, TIFF0007641371000042.tif718 can for example be calculated directly by changing the variables: TIFF0007641371000043.tif783(10) Where: TIFF0007641371000044.tif737 defines the Jacobian consisting of all first order partial derivatives.

[0215] For example, in each of the devices 500, 600, and 800, the function TIFF0007641371000045.tif77 (e.g., training flow block 510 1...N ) is executed in each device 100, 200, 400 by the function TIFF0007641371000046.tif74 (e.g., inference flow block 110 1...N (composed of partial functions executed by The function definition in TIFF0007641371000047.tif74, for example, is invertible because the partial function performed by the training flow block is The function definition in TIFF0007641371000048.tif77 may be derived. Thus, the functions executed by the training devices 500, 600, 800 By determining the rules that define TIFF0007641371000049.tif74 (e.g., neural network parameters), the function The definition of TIFF0007641371000050.tif79 is also implicitly obtained.

[0216] In other words, The function definition of TIFF0007641371000051.tif77 may be determined in training (e.g., by determining neural network parameters such that a training audio signal x is transformed into a noise like signal z), The function definition for TIFF0007641371000052.tif74 is: It may be derived from the function definition in TIFF0007641371000053.tif77.

[0217] In the following, some further (optional) details regarding the speech enhancement flow, which may optionally be used in the apparatus and method according to embodiments of the present invention (e.g. in the apparatus 100, 200, 400, 500, 600, 800 or in the flow blocks 311, 711), are described.

[0218] For speech enhancement (or, in general, for audio enhancement), we have a time-domain mixture signal of length N TIFF0007641371000054.tif718 is a clean speech recording TIFF0007641371000055.tif718 and some additional interfering background TIFF0007641371000056.tif718, where, for example, the noisy mixture is shown as the sum of clean speech and an interfering background, resulting in y = x + n(11).

[0219] moreover, TIFF0007641371000057.tif718 is defined as being sampled from a normal distribution with zero mean and unit variance, e.g. as a Gaussian sample, i.e. z~N(z;0,I)(12).

[0220] The flow block-based model proposed in the devices 500, 600 and 800 shown in Figs. 5, 6 and 8 is defined as a DNN, which estimates the probability distribution p formed by clean speech utterance x conditioned on a noisy mixture y. x The objective is to approximate (x|y), e.g., to learn the probability distribution function of x conditional on y. Minimizing the negative log-likelihood of a predefined probability distribution can, for example, be considered as a training objective (e.g., optimization of neural network parameters may minimize the value of the following expression): TIFF0007641371000058.tif797(13) Here, θ represents a neural network parameter.

[0221] In the enhancement step (e.g., in the apparatuses 100, 200 and 400 shown in Figs. 1, 2 and 4), the sample is z (z) and passed as input to the neural network together with the noisy sample, e.g. sample z follows an inverted flow with the noisy input y. For example, time-domain sample values ​​of a noise-like signal having a predetermined distribution, e.g. a Gaussian distribution, may be input to the (first) flow block (and thus, e.g. in a pre-processed form, e.g. to the neural net together with samples of audio signal y). Following the inverted flow, e.g. the inverted flow block processing reversed to the processing performed in the training flow block, the neural network (e.g. in combination with the affine processing 372) maps the random sample (or samples) back to the distribution of the clean utterance and generates an enhanced audio signal, e.g. ideally the underlying TIFF0007641371000059.tif73, for example Close to TIFF0007641371000060.tif711 Enhanced speech signal with TIFF0007641371000061.tif73 Generates TIFF0007641371000062.tif73.

[0222] These correspondences 7 to 13 are also valid for the system 1000 shown in FIG.

[0223] The following describes non-linear input compression, eg, compression and / or expansion, that may optionally be used in embodiments of the present invention.

[0224] FIG. 9 shows a diagram of the non-linear input compression step used in the apparatus described herein.

[0225] The non-linear input compression step may correspondingly be used as a pre-processing block in, for example, any of the devices 500, 600 or 800 shown in FIG.

[0226] A non-linear compression algorithm may be applied to map small amplitudes of the audio data samples to wider intervals and larger amplitudes to smaller intervals.

[0227] Fig. 9 shows at 910 an audio signal, e.g. a clean signal x, shown as a function of time, e.g. in a time domain representation. The speech signal shows an example of a speech signal, e.g. x, on which a neural network associated with the device 500, 600 or 800 shown in Fig. 5, 6 or 8 is trained. Since the neural network models the probability distribution of the time domain audio, e.g. speech, utterance, it is important to examine the range of values ​​over which the distribution is trained. Audio data is typically stored as a normalized 32-bit flow, e.g. in the range [-1, 1]. Time domain audio, e.g. speech, samples approximately follow a Laplacian distribution.

[0228] The audio, e.g. speech signal, is shown with histograms of values ​​before (a) and after (b) application of a compression, algorithm, e.g. nonlinear input compression, e.g. 815. Compression may be understood, for example, as a kind of histogram equalization, or as a histogram spreading of relatively low signal values ​​and / or a histogram compression of relatively large signal values. As can be seen, for example, from a comparison of a first histogram 920 before application of a compression algorithm and a second histogram 920 after application of a compression algorithm, it can be seen that the histograms become wider. For example, the horizontal axis 922 of the histogram 920 shows the signal values ​​before compression, and the vertical axis 924 shows the probability distribution of each signal value. For example, the horizontal axis 932 of the histogram 920 shows the signal values ​​after compression, and the vertical axis 934 shows the probability distribution of each signal value. It becomes clear that the compressed signal values ​​contain a wider (more evenly distributed, less peak-like) probability distribution, which is known to be advantageous for processing in the flow block.

[0229] As shown in Fig. 9(a), (e.g., reference numeral 920), most of the values ​​of the approximate Laplacian distribution are in a small range around 0. As seen in Fig. 9(a), it is recognized that in a clean speech sample (or clean speech signal), e.g., x, data samples (or signal values) with higher absolute amplitudes carry significant information and are usually under-represented. When compression is applied, the algorithm provides that the values ​​of the time-domain speech samples are more evenly spread.

[0230] In an embodiment, the non-linear input compression may be represented by, for example, μ-law compression of the input data without quantization: TIFF0007641371000063.tif1347(14) where sgn() is the sign function, μ is a parameter that defines the compression level.

[0231] The parameter μ may be set, for example, to 255, a common value used in telecommunications.

[0232] Regarding learning, e.g., training targets, in the flow block processing of the device 500, 600 or 800 shown in FIG. 5, FIG. 6 or FIG. 8, instead of learning the distribution of a clean utterance, e.g., an unprocessed clean signal x, the distribution of a compressed signal, e.g., a preprocessed signal x, is learned.

[0233] The described nonlinear input compression algorithm and its inverse algorithm may be used, for example, in the devices 100, 200 or 400 shown in Figures 1, 2 and 4 as a final processing step, for example the nonlinear expansion 415 shown in Figure 4. The highlighted samples of Figures 1, 2 and 4 TIFF0007641371000064.tif73 can be expanded to a regular signal, for example by using an inverting μ-law transform, e.g.: TIFF0007641371000065.tif1354(15) where sgn() is the sign function, μ is a parameter that defines the expansion level.

[0234] However, it should be noted that the non-linear input compression shown in FIG. 9 may be optionally supplemented by any of the features, functions and details disclosed herein, either individually or in combination.

[0235] FIG. 10 shows a schematic diagram of a flow block system 1000 for audio signal processing according to an embodiment.

[0236] Flow block system 1000 represents a combination of apparatus 1100 for providing neural network parameters and apparatus 1200 for providing a processed signal (which may also be used individually). Apparatus 1100 may be implemented, for example, as any of apparatuses 500, 600 or 800 shown in Figures 5, 6 or 8. Apparatus 1200 may be implemented, for example, as any of apparatuses 100, 200 or 400 shown in Figures 1, 2 or 4.

[0237] In device 1100, the clean-noise pair (xy) is subjected to (or is input to) a flow block process to be mapped to a Gaussian distribution N(z;0;I) (or a distribution that approximates a Gaussian distribution). In inference (device 1200), a sample z (e.g., a block of sample values) is drawn from this distribution (or from a signal having a desired, e.g., Gaussian, distribution of signal values) and followed by an inverse flow block process with another noisy utterance y to produce an enhanced signal Generates TIFF0007641371000066.tif73.

[0238] The apparatus 1100 is configured to provide neural network parameters based on a training audio signal 1105, e.g., a clean x, e.g., x1, and a distorted version of the training audio signal 1130, e.g., a noisy y, e.g., y1=x1+n1. The process includes a number of N flow blocks, e.g., training flow block 1010, associated with the neural network (not shown). 1...N The flow block 1010 executes the 1...N is configured to process an incoming audio signal, for example a speech signal.

[0239] Before introducing the training audio signal 1105 into the first flow block 11101 of the apparatus 1100, the audio samples of the training audio signal x are grouped 1116 into groups of samples, for example groups of 8 samples, for example vectors.

[0240] An arbitrarily grouped training audio signal x(i=1), say x1(i=1), is introduced into a first flow block 11101 of the apparatus 1100 together with a distorted training audio signal y, 1130, say y1.

[0241] The distorted audio signal y, 1130 is, for example, a noisy input signal. The distorted training audio signal y, 1130 is, for example, defined as y=x+n, where x is a clean part of the input signal, for example the training input signal x, 1105, and n is the noisy background, for example y1=x1+n1, where x1 is a clean part of the input signal, for example the training input signal x1, 1105, and n1 is the noisy background. The distorted training audio signal y, 1130 may, for example, be represented as time-domain audio samples, for example noisy time-domain speech samples.

[0242] The training audio signal x and the corresponding distorted versions of the training audio signal y may optionally be grouped into a vector representation (or a matrix representation).

[0243] The apparatus 1100 comprises a training flow block 1110 to be mapped to a distribution, e.g. a Gaussian distribution, of a training result audio signal 1120, e.g. a noise signal, e.g. z. 1...N Based on the clean-noise (xy) pair (or based on a number of clean-noise pairs) that follows (or is processed by) a neural network (e.g., neural network 580 1...N ) The training result audio signals 1120 may optionally be grouped into a vector representation (or a matrix representation).

[0244] A training audio signal x, 1105 is introduced into a first flow block 11101 of the apparatus 1100 together with a distorted training audio signal y, 1130. The training audio signal x, 1105 is, for example, represented as audio samples, for example as time domain samples.

[0245] Flow Block 1110 1...N may be, for example, the flow block 510 of the device 500, 600 or 800 shown in FIG. 1...N , 610 1...N , or 810 1...N It may be implemented as:

[0246] Flow Block 1110 1...N may include an affine connection layer block, such as flow block 6111, or 7111, or 8111, as shown in, for example, FIGS.

[0247] Flow Block 1110 1...N As an output of the training result audio signal z, 1120 is provided, which is for example a noise signal (or an approximation to a noise signal). The noise signal z, 1120 is for example defined as z ≠ N (z; 0; I).

[0248] In the apparatus 1100, a clean signal x, 1105 is fed to a training flow block 1110. 1...N, is introduced along with a corresponding distorted, e.g., noisy, audio signal y, 1130 to train a neural network associated with the neural network and determine neural network parameters, e.g., edge weights (θ), of the neural network as a result of the training.

[0249] The neural network parameters determined by apparatus 1100 may be further used, for example, in the inference provided by apparatus 1200.

[0250] The apparatus 1200 is configured to provide a processed, e.g. enhanced, audio signal based on an input audio signal y, 1230. The processing is carried out by N flow blocks associated with a neural network (not shown), e.g. an inference flow block 1210 1...N The flow is executed in flow block 1210. 1...N is configured to process an incoming audio signal, for example a speech signal.

[0251] An input audio signal y, 1230, e.g. a new noisy signal y2, is introduced into the device 1200 to be processed. The input audio signal y is e.g. a noisy input signal or e.g. a distorted audio signal. The input audio signal y, 1230 is defined as y=x+n, where x is the clean part of the input audio signal and n is the noisy background, e.g. y2=x2+n2. The input audio signal y, 1230 may be represented as e.g. time-domain audio samples, e.g. noisy time-domain speech samples.

[0252] The input audio signal y and its correspondingly cleaned portion x may be arbitrarily grouped into a vector representation (or matrix representation).

[0253] A noise signal z, 1220 is obtained (e.g., generated) and introduced into a first flow block 12101 of the apparatus 100 together with an input audio signal y, 1230. The noise signal z, 1220 is defined, for example, as sampled from a normal distribution with zero mean and unit variance, e.g., z ∼ N(z; 0; I). The noise signal z, 1220 is represented, for example, as noise samples, e.g., time-domain noise samples.

[0254] Before introducing the noise signal 1220 into the first flow block 12101 of the apparatus 1200, the audio samples of the noise signal z are optionally grouped 1216 into groups of samples, for example groups of 8 samples, for example into vectors (or into matrices). This grouping step may, for example, be arbitrary.

[0255] An arbitrary grouped noise signal z(i=1), e.g. x1(i=1), is introduced into the first flow block 12101 of the device 1200 together with an input audio signal y, 1230, e.g. y2. 1...N The flow block 1110 of the device 1100 1...N (eg, compared to the corresponding flow block of the apparatus 1100, which performs an inverse affine operation and optionally also a deconvolution operation).

[0256] Flow Block 1210 1...N may be, for example, the flow block 110 of the device 100, 200 or 400 shown in FIG. 1...N , 210 1...N , or 410 1...N It may be implemented as:

[0257] Flow Block 1210 1...N may include an affine connection layer block, such as flow block 2111, or 3111, or 4111, as shown in, for example, FIGS.

[0258] Flow Block 1210 1...NAs an output of TIFF0007641371000067.tif73, 1260 is provided. Enhanced audio signal TIFF0007641371000068.tif73,1260, for example, represents the enhanced clean portion of the input audio signal,y,1230.

[0259] The clean portion x of the input audio signal y, 1230 is not introduced separately to the apparatus 1200. The apparatus 1200 processes, e.g., a generated noise signal z, 1220 based on the input audio signal y, 1230, to receive, e.g., generate, e.g., output, an enhanced audio signal, which is, e.g., an enhancement of the clean portion of the input audio signal y, 1230.

[0260] However, it should be noted that the system 1000 may be optionally supplemented with any of the features, functions and details disclosed herein, either individually or in combination.

[0261] FIG. 11 shows Table 1 showing a comparison between the device and method according to the embodiment and the prior art.

[0262] Table 1 shows the evaluation results using the objective evaluation indices. SE-Flow represents the proposed flow-based approach according to the embodiments as described above with reference to Figs. 1-3, 5-7 and 10, and SE-Flow-μ represents the approach including μ-law transformation as a corresponding pre- or post-processing step, e.g. as a nonlinear compression, e.g. as compression described with reference to Fig. 9, e.g. as a nonlinear input compression or a nonlinear expansion, and correspondingly as used in the embodiments described above with reference to Figs. 8 and 4.

[0263] As shown in the table, between the two proposed flow-based experiments, the model using μ-compression shows better results in all metrics, demonstrating the effectiveness of this easy pre- and post-processing technique for modeling the distribution of time-domain signals, e.g., nonlinear compression.

[0264] An illustration of the highlighting capabilities can also be seen in FIG.

[0265] FIG. 12 shows a graphical representation illustrating the performance of an apparatus and method according to an embodiment.

[0266] Fig. 12 shows exemplary spectrograms to illustrate the performance of the proposed embodiment. In (a) a loud speech utterance of 2.5 dB (signal-to-noise ratio, SNR) is displayed. In (b) the corresponding clean speech is shown. In (c) and (d) the results of the proposed flow-based system according to an embodiment of the present invention, e.g. as shown in Fig. 10, are shown.

[0267] Further embodiments and aspects Further aspects and embodiments according to the present invention are described below, which can be used individually or in combination with any other embodiment disclosed herein.

[0268] Furthermore, the embodiments disclosed in this section may be optionally supplemented with any other features, functions, and details disclosed herein, both individually and in any combination.

[0269] In the following, we present the concept of flow-based neural networks for time-domain speech enhancement.

[0270] In the following, the basic embodiment concepts of the present invention are explained.

[0271] Below, certain goals and objectives of the present invention that may be achieved (at least partially) in some or all of the embodiments are described and certain aspects of the present invention are briefly summarized.

[0272] Speech enhancement involves distinguishing a target speech signal from an intrusive background. While traditional generative approaches using variational autoencoders or generative adversarial networks (GANs) have been increasingly used in recent years, normalized flow (NF)-based systems remain scarce despite their success in related fields. Therefore, in the following, a NF framework is proposed according to an embodiment to directly model the enhancement process by density estimation of clean speech utterances conditioned on a noisy opponent. Traditional models inspired by speech synthesis are adapted in an embodiment to enable direct enhancement of noisy utterances in the time domain. Experimental evaluation on publicly available datasets according to an embodiment shows comparable performance to current state-of-the-art GAN-based approaches while exceeding baselines selected using objective evaluation metrics.

[0273] Embodiments of the present invention can be used for speech enhancement, utilizing flow normalization and / or deep learning and / or generative modeling.

[0274] A brief introduction is provided below.

[0275] Traditionally, the aim of speech enhancement (SE) is to enhance a target speech signal from an interfering background to ensure better intelligibility of the speech content [1]. Due to its importance for a wide range of applications including, for example, hearing aids [2] or automatic speech recognition [3], it has been extensively studied in the past. In doing so, deep neural networks (DNNs) have largely replaced traditional techniques such as Wiener filtering [4], spectral subtraction [5], subtraction methods [6] or minimum mean square error (MMSE) [7]. Most commonly, DNNs are traditionally used to estimate time-frequency (TF) masks capable of separating speech and background from a mixed signal [8]. Nevertheless, systems based on time-domain inputs have been proposed in recent years [9, 10, 11], which have the advantage of avoiding the expensive TF transform. Recently, there has also been growing attention in SE research on generative approaches such as generative adversarial networks (GANs) [11, 12, 13], variational autoencoders (VAEs)

[14] , and autoregressive models

[10] . In particular, the use of GANs, in which a generator and a discriminator are simultaneously trained adversarially, has been widely investigated in the past few years. For example, Pascual et al.

[11] proposed a GAN-based end-to-end system in which a generator directly enhances noisy speech samples at the waveform level. In the following, this approach has been extended multiple times, for example, by using the Wasserstein distance

[15] or by combining multiple generators to improve performance

[16] . Other researchers have reported impressive SE results working with GANs to estimate clean TF spectrograms by implementing additional techniques such as mean squared error regularization

[12] or by directly optimizing the network with respect to speech-specific evaluation metrics

[13] . Although the aforementioned traditional approaches have become increasingly popular recently, normalized flow (NF)-based systems are still rare in SE. Very recently, the work of Nugraha et al.

[17] proposed a flow-based model combined with VAE to learn deep latent representations that can be used prior to deep speech recognition.However, those approaches do not model the enhancement process itself and therefore depend on the SE algorithm with which it is combined. However, it has been shown in areas such as computer vision

[18] or speech synthesis

[19] that NFs have the ability to successfully generate high-quality samples in the respective tasks. This therefore leads to the assumption of the underlying embodiment of the present invention that enhancement of speech samples can be directly performed using a flow-based system by modeling the generative process.

[0276] The concept of the present invention is that NF can be successfully applied to SE by learned mapping from easy probability distributions based on clean speech samples conditioned on noisy opponents to more complex probability distributions. Thus, in the present invention, a conventional flow-based DNN architecture is modified from speech synthesis to perform SE directly in the time domain without the need for predefined features or TF transformation. Furthermore, in the present invention, an easy pre-processing technique of the input signal using compression, e.g., nonlinear compression, as part of the compression process is applied to enhance the performance of the SE model based on density estimation. Experimental evaluation of the proposed method and apparatus of the present invention confirms these assumptions and shows an improvement over or surpasses the results of other time-domain GAN baselines compared to the current state-of-the-art systems.

[0277] Figure 10 shows an overview of the proposed system according to an embodiment. A clean-noisy (xy) pair follows a flow step (solid blue line) that is mapped to a Gaussian distribution N(z;0,I). Inference, a sample z is drawn from this distribution and follows the inverse flow (dashed red line) with another noisy utterance y to produce an enhanced signal. Generate TIFF0007641371000069.tif73 (take into account the different hatching seen in colors or in the block contours).

[0278] Problem Statement and Aspects of the Embodiments In the following, a systematic discussion and some explanations of the issues related to aspects of embodiments of the present invention are provided.

[0279] Normalization flow basics Normalized flow can be described as a transformation from a simple probability distribution to a more complex one using a reversible and differentiable mapping, formally expressed as follows

[20] :

[0280] x = f(z) (16) Where: TIFF0007641371000070.tif714 and TIFF0007641371000071.tif713 is a D-dimensional random variable, TIFF0007641371000072.tif73 is a functional mapping from z to x. The invertibility of TIFF0007641371000073.tif73 ensures that this step can be inverted back from x to z, i.e. TIFF0007641371000074.tif722(17).

[0281] In addition, the function If TIFF0007641371000075.tif73 is invertible and differentiable, then the construction of a sequence of 1 to T transforms is guaranteed to be invertible as well. TIFF0007641371000076.tif719... TIFF0007641371000077.tif712(18) This is followed by the log probability density function TIFF0007641371000078.tif718 can be calculated by changing the variables

[21] : TIFF0007641371000079.tif783(19) Where: TIFF0007641371000080.tif737 defines the Jacobian consisting of all first order partial derivatives.

[0282] Speech emphasis flow For speech enhancement according to an embodiment of the present invention, a time-domain mixture signal of length N TIFF0007641371000081.tif718 is a clean speech recording TIFF0007641371000082.tif719 with some additional distracting background TIFF0007641371000083.tif718 and the result is y = x + n(20).

[0283] Additionally, it is sampled from a normal distribution with zero mean and unit variance. TIFF0007641371000084.tif718, i.e. z~N(z;0,I)(21).

[0284] The proposed NF model according to an embodiment of the present invention is defined here as a DNN, which estimates the probability distribution p formed by a clean speech utterance x conditioned on a noisy mixture y. x The aim is to outline (x|y). As a training objective according to an embodiment of the present invention, the negative log-likelihood of the previously defined probability distribution, for example of speech samples, can now be simply minimized: TIFF0007641371000085.tif797(22) Here, θ represents a network parameter. TIFF0007641371000086.tif727 is, for example, a probability distribution for speech samples to be defined, TIFF0007641371000087.tif734 is, for example, the likelihood of a Gaussian function, TIFF0007641371000088.tif727 shows the level (eg, how much to vary) to which the Gaussian function is varied to create, eg, generate, a speech sample.

[0285] In the highlighting step, according to an embodiment of the present invention, this is p z (z) can be passed as input to the network along with the noisy samples. Following the inversion flow, the neural network maps the random samples back to the distribution of clean utterances, ideally refining the underlying x, e.g. Close to TIFF0007641371000089.tif711 TIFF0007641371000090.tif73, This creates TIFF0007641371000091.tif73. This process is also shown in Figure 10.

[0286] In practice, for example, in modeling or in neural networks, the total amount of neural network parameters may be, for example, According to an embodiment of the present invention, during training of a neural network, signals x, y are introduced into the neural network to be trained. The output of the neural network is z (the processed signal after all flow blocks), TIFF0007641371000093.tif711 (from each affine connection layer) and Contains TIFF0007641371000094.tif720 (from each 1x1 lossless convolution).

[0287] The loss function to be optimized is TIFF0007641371000095.tif1310+ TIFF0007641371000096.tif713+ TIFF0007641371000097.tif724=number / scalar(23), TIFF0007641371000098.tif1310 is the likelihood of the Gaussian function (see above), ( TIFF0007641371000099.tif713+ TIFF0007641371000100.tif724)-( TIFF0007641371000101.tif718 (see above).

[0288] A method proposed according to an embodiment of the present invention Model Architecture According to an embodiment of the present invention, the Waveglow architecture

[19] has been modified for speech synthesis to perform speech enhancement. Essentially, the model takes a speech utterance together with its corresponding mel-spectrogram as input for several steps of the flow and learns to generate realistic speech samples based on the input conditioned spectrograms. One flow block consists of a 1×1 reversible convolution

[22] that ensures the exchange of information along the channel dimension, and a so-called affine connection layer

[23] that is used to ensure the invertibility and efficient computation of the Jacobian determinant. Thus, the input signal is split along the channel dimension and one half is fed into a Wavenet-like NN block, e.g., the Wavenet line affine connection layer, which defines the scaling and translation coefficients of the latter half. To create this multi-channel input, multiple audio samples are stacked together in one group, mimicking a multi-channel signal. The affine connection layer is also where the conditioned information is included. For further details of this procedure, see

[19] . As the original Waveglow is computationally heavy (>87 Mio. parameters), several architectural modifications were made to make it feasible to train it on a single GPU and enable enhanced speech. In contrast to Waveglow, rather than Mel-spectrograms, noisy time-domain speech samples were used as conditioned input according to the present embodiment. Hence, no upsampling layers were required as both signals are of the same dimensionality. Furthermore, standard convolutions in Wavenet-like blocks were replaced by depthwise separable convolutions

[24] to reduce the amount of parameters, as recommended in

[25] , according to the present embodiment.

[0289] Nonlinear Input Compression FIG. 9 shows an example of the effect of non-linear input compression, e.g., compression, according to an embodiment of the present invention. At the top, a clean speech utterance is shown. (a) is the histogram (n bins = 100). In (b) we can see the effect of compression algorithms such as compression on the values.

[0290] Since the network models the probability distribution of time-domain speech utterances, it is important to examine the range of values ​​over which the distribution is to be learned. The audio data was stored as normalized 32-bit floats in the range [-1,1]. Since time-domain speech samples approximately follow a Laplacian distribution

[27] , it is easy to see that most of the values ​​are in a small range around zero (see Fig. 9(a)). However, especially in clean speech utterances, data samples with higher absolute amplitudes carry important information and are not well represented in this case. To ensure that the values, e.g. the learnable amplitude values, are more evenly spread, a non-linear compression, e.g. a compression algorithm, according to an embodiment of the present invention can be applied to map small amplitudes to wider amplitudes and large amplitudes to smaller intervals. This is illustrated in Fig. 9, where one speech sample is displayed together with a histogram of values ​​before and after applying the compression, e.g. a compression algorithm. In this sense, the compression, e.g. compression, can be understood as a kind of histogram equalization. Following this, we conducted additional experiments using μ-law compression of the input data without quantization, e.g., compression (ITU-T Recommendation G.711), which is formally defined as follows: TIFF0007641371000102.tif1347(24) where sgn() is the sign function and μ is a parameter that defines the compression level. Here, according to an embodiment of the present invention, μ is set to 255 throughout the experiments, which is a common value also used in telecommunications. For the learning purpose, instead of learning the distribution of clean speech, the distribution of the compressed signal is learned. The enhanced samples can be expanded to the normal signal after inverting the μ-law transform.

[0291] experiment data The dataset used in the experiments was published with the work of Valentini et al.

[28] and is a commonly used database for the development of SE algorithms. It contains 30 individual speakers from the Voice Bank corpus

[29] separated into training and test sets with 28 and 2 speakers, respectively. Both sets are balanced according to male and female participants. The training samples were mixed with 8 real noise samples from the DEMAND database

[30] and 2 artificial (bubbly and speech-shaped) samples with signal-to-noise ratios (SNR) of 0, 5, 10, and 15 dB. In the test set, different noise samples were selected and mixed according to SNR values ​​of 2.5, 7.5, 12.5, and 17.5 dB, so that the test set included only unseen conditions. In addition, one male and one female speaker were taken from the training set to create a validation set for model development.

[0292] Training Strategy Batch Size TIFF0007641371000103.tif73[4,8,12], Number of flow blocks TIFF0007641371000104.tif73[8,12,16], and the amount of samples grouped together as input Values ​​of TIFF0007641371000105.tif73[8,12,24] were chosen in a small hyperparameter search, for example. Each individual model was trained for 150 epochs to select parameters based on the lowest validation loss. Some initial experiments using higher batch sizes were performed, but it was found that the models did not generalize well. The selected models were further trained until convergence based on a robust early stopping mechanism for 20 epochs. A fine-tuning step followed, using a reduced learning rate and the same early stopping criterion.

[0293] Model Settings As a result of the parameter search, we built a model with 16 flow blocks, groups of 12 samples as input, and a batch size of 4. The learning rate was 3 × 10 in the initial training step using the Adam

[31] optimizer and weight regularization

[32] . -4 For fine tuning, the learning rate was set to 3×10 -5 For the training input, we used 1s chunks (sampling rate TIFF0007641371000106.tif74 = 16 kHz) was randomly extracted from each audio file. The standard deviation of the Gaussian distribution was set to σ = 1.0. As with other NF models

[33] , the effect of using a lower value of σ in inference was experienced to obtain a higher quality output, which is why σ was set to σ = 0.9 in inference. According to the Wavenet-like block of the original Waveglow architecture with 8 layers of dilated convolutions, 512 channels were used as residual connections and 256 channels in skip connections for the affine connection layer. In addition, for each of the four connection layers, two channels were passed to the loss function to form a multi-scale architecture.

[0294] evaluation To compare approaches according to embodiments of the present invention with current work in the field, the following evaluation metrics were used: · (i) Perceptual Evaluation of Voice Quality (PESQ) in the recommended wideband version of ITU-T P.862.2 (-0.5 to 4.5). Three mean opinion score (1 to 5) indices

[34] : (ii) prediction of signal distortion (CSIG), (iii) prediction of background intrusiveness (CBAK), and (iv) prediction of overall speech quality (COVL). Improved partial SNR (segSNR)

[35] (from 0 to TIFF0007641371000107.tif74).

[0295] As a baseline for the proposed method according to the embodiment of the present invention, two generative time-domain approaches, namely SEGAN

[11] and Improved Deep SEGAN (DSEGAN)

[16] models, were defined since they were evaluated by the same database and metrics. Furthermore, we compared it with two other state-of-the-art GAN-based systems operating on TF masks, namely MMSE-GAN

[26] and Metric-GAN

[13] . It is noted that there are several discriminative approaches, e.g. [9, 36, 37], that report higher performance on this dataset. However, since the focus of this work was on generative models, they are not included in the comparison.

[0296] Experimental Results The experimental results are displayed in Table 1, shown in Fig. 11. Table 1 shows the evaluation results using objective evaluation metrics. SE-Flow represents the proposed flow-based approach, and SE-Flow-μ represents the approach together with μ-law compression, which includes, for example, compression and expansion of the input data. The values ​​of all the compared methods are taken from the corresponding papers.

[0297] As shown in the table, between the two proposed flow-based experiments, the model using μ-compression, including compression and expansion, shows better results in all metrics. This demonstrates the effectiveness of this easy pre-processing and corresponding post-processing technique for modeling the distribution of time-domain signals. The explanation of the highlighting function can also be seen in Figure 12.

[0298] Fig. 12 shows exemplary spectrograms to illustrate the performance of the proposed system. In (a) a loud speech utterance of 2.5 dB (signal-to-noise ratio, SNR) is displayed. In (b) the corresponding clean speech is shown. In (c) and (d) the results of the proposed flow-based system according to an embodiment of the present invention are shown.

[0299] Comparing the spectrograms of the two proposed systems according to the present invention, SE-Flow-μ seems to be able to capture finer speech parts with less background leakage. Also, note that the breath sounds at the end of the displayed example are not recovered by the model according to the present invention, which emphasizes that the proposed model according to the present invention focuses on real speech samples. Furthermore, in the flow-based example, we can see that there is more noise-like frequency content at higher frequencies when speech is active compared to the clean signal. This can be explained by the Gaussian sampling, which is not completely eliminated during inference.

[0300] Compared to the SEGAN baseline, the proposed method and apparatus according to the embodiments of the present invention show superior performance across all metrics by a large margin. Note that only for SEGAN, we can look at the segSNR performance, since it was not evaluated for the other methods. Looking at DSEGAN, we can see that the proposed SE-Flow reaches the compared performance in CSIG, while showing slightly lower values ​​in other metrics. However, the SE-Flow-μ-based system or method or apparatus according to the embodiments of the present invention still performs better in all metrics except COVL. Thus, within the time domain approach, the proposed flow-based model according to the embodiments of the present invention seems to better model the generative process from noisy signals to enhanced signals. With regard to MMSE-GAN, it is observed that this approach has a similar performance with a slight edge to MMSE-GAN, although no additional regularization techniques are implemented here. However, Metric-GAN shows superior results compared to the proposed approach for all metrics displayed. However, it is important to note that the good performance of the present embodiment is expected because the model according to the present embodiment was directly optimized according to the PESQ metric. As a result, coupling training with direct optimization of the evaluation metric may also be an effective way to improve the system or method or apparatus according to the present embodiment.

[0301] conclusion In this disclosure, a normalized flow-based speech enhancement method according to an embodiment of the present invention is introduced. The model according to an embodiment of the present invention enables signal enhancement by density estimation and generative inference of clean speech samples given a noisy opponent. Simple nonlinear compression, e.g., compression or expansion techniques according to an embodiment of the present invention have been demonstrated to be effective (optional) pre-processing or e.g., post-processing tools to increase enhancement results. The proposed system and method and apparatus according to an embodiment of the present invention outperform other time-domain GAN-based baselines while approaching state-of-the-art TF techniques. According to an embodiment of the present invention, exploring different techniques in the coupling layer as well as the combination of time-domain and frequency-domain signals can be further implemented.

[0302] Furthermore, it should be noted that the embodiments and procedures may be used as described in this section (as well as in the sections "Formulation of the Problem", "Fundamentals of Normalization Flow", "Speech Enhancement Flow", "Proposed Methodology", "Model Architecture", "Nonlinear Input Compression", "Experiments", "Data", "Training Strategy", "Model Settings", "Evaluation" and "Experimental Results") and may be optionally supplemented by any of the features, functions and details disclosed herein, both individually and in combination.

[0303] However, features, functions and details described in any other section may also be optionally incorporated into embodiments according to the present invention.

[0304] Also, the embodiments described in the above sections can be used individually or can be supplemented with any of the features, functions and details of the other sections.

[0305] It should also be noted that the individual aspects described herein can be used individually or in combination, and thus details can be added to each of the individual aspects without adding the details to another one of the aspects.

[0306] In particular, the embodiments are also set forth in the claims, which can be optionally supplemented with any of the features, functions and details described herein, both individually and in any combination.

[0307] Moreover, features and functions disclosed herein in relation to a method may also be used in an apparatus (configured to perform such functions). Moreover, any features and functions disclosed herein in relation to an apparatus may also be used in the corresponding method. In other words, the methods disclosed herein may be supplemented by any of the features and functions described in relation to the apparatus.

[0308] Additionally, any of the features and functions described herein can be implemented in hardware or software, or using a combination of hardware and software, as described in the "Implementation Alternatives" section.

[0309] Further concluding, embodiments of the present invention form a normalized flow (NF)-based system for use in the speech enhancement field (e.g., building a speech enhancement framework using normalized flows that directly model the enhancement process, including, for example, a learned probability distribution for clean speech).

[0310] To further conclude, embodiments of the present invention form the concept of a flow-based system applied to speech enhancement, in particular by using a flow-based system to directly perform audio signal enhancement independent of other algorithms and combining the resulting signal without compromising the audio signal enhancement performance and quality.

[0311] Furthermore, embodiments according to the present invention provide a trade-off between efficient modeling of flow-based audio signal processing using neural networks and audio signal enhancement capabilities.

[0312] Implementation Alternatives Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, with blocks or apparatus corresponding to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or function of a corresponding apparatus. Some or all of the method steps can be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps can be performed by such an apparatus.

[0313] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be done using digital storage media such as floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs, EPROMs, EEPROMs, flash memories, etc., on which electronically readable control signals are stored and which cooperate (or can cooperate) with a programmable computer system to execute the respective methods. Thus, the digital storage media can be computer readable.

[0314] Some embodiments of the present invention comprise a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0315] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which operates to perform one of the methods when the computer program product is run on a computer. The program code may for example be stored on a machine readable carrier.

[0316] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0317] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0318] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) comprising recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0319] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, the data stream or the sequence of signals may be adapted to be transferred via a data communication connection, such as for example the Internet.

[0320] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0321] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0322] Further embodiments according to the invention comprise an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0323] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0324] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0325] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0326] The methods described herein can be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0327] Any of the methods described herein, or components of the apparatus described herein, may be at least partially implemented by hardware and / or software.

[0328] The embodiments described herein are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as descriptions and illustrations of the embodiments herein.

[0329] References: [1] P Loizou, Speech Enhancement: Theory and Practice, CRC Press, 2 edition, 2013.

[0330] [2] K. Borisagar, D. Thanki, and B. Sedani, Speech Enhancement Techniques for Digital Hearing Aids, Springer International Publishing, 2018.

[0331] [3] AH Moore, P. Peso Parada, and PA Naylor, “Speech enhancement for robust automatic speech recognition: Evaluation using a baseline system and instrumental measures,” Computer Speech & Language, vol. 46, pp. 574 - 584, 2017.

[0332] [4] J. Lim and A. Oppenheim, “All-pole modeling of degraded speech,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 26, no. 3, pp. 197-210, 1978.

[0333] [5] S. Boll, “Suppression of acoustic noise in speech using spectral subtraction,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 27, no. 2, pp. 113-120, 1979.

[0334] [6] Y. Ephraim and H.L. Van Trees, “A signal subspace approach for speech enhancement,” IEEE Transactions on Speech and Audio Processing, vol. 3, no. 4, pp. 251-266, 1995.

[0335] [7] Y. Ephraim and D. Malah, “Speech enhancement using a minimum mean-square error log-spectral amplitude estimator,” IEEE Transactions Acoustics, Speech and Signal Processing, vol. 33, pp. 443 - 445, 05 1985.

[0336] [8] Y. Xu, J. Du, L. Dai, and C. Lee, “A regression approach to speech enhancement based on deep neural networks,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 1, pp. 7-19, 2015.

[0337] [9] F. Germain, Q. Chen, and V. Koltun, “Speech denoising with deep feature losses,” in Proc. Interspeech Conf., 2019, pp. 2723-2727.

[0338]

[10] K. Qian, Y. Zhang, S. Chang, X. Yang, D. Florencio, and M. Hasegawa-Johnson, “Speech enhancement using bayesian wavenet,” in Proc. Interspeech Conf., 2017, pp. 2013-2017.

[0339]

[11] S. Pascual, A. Bonafonte, and J. Serra`, “Segan: Speech enhancement generative adversarial network,” in Proc. Inter-speech Conf., 2017, pp. 3642-3646.

[0340]

[12] M. H. Soni, N. Shah, and H. A. Patil, “Time-frequency masking-based speech enhancement using generative adversarial network,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5039-5043.

[0341]

[13] S.-W. Fu, C.-F. Liao, Y. Tsao, and S.-D. Lin, “Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement,” in Proc. Intl. Conf. Ma-chine Learning (ICML), 2019, pp. 2031-2041.

[0342]

[14] S. Leglaive, X. Alameda-Pineda, L. Girin, and R. Horaud, “A Recurrent Variational Autoencoder for Speech Enhancement,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 371-375.

[0343]

[15] N. Adiga, Y. Pantazis, V. Tsiaras, and Y. Stylianou, “Speech enhancement for noise-robust speech synthesis using wasserstein gan,” in Proc. Interspeech Conf., 2019, pp. 1821-1825.

[0344]

[16] H. Phan, I. V. McLoughlin, L. Pham, O. Y. Chen, P. Koch, M. De Vos, and A. Mertins, “Improving gans for speech enhancement,” IEEE Signal Processing Letters, vol. 27, pp. 1700-1704, 2020.

[0345]

[17] A. A. Nugraha, K. Sekiguchi, and K. Yoshii, “A flow-based deep latent variable model for speech spectrogram modeling and enhancement,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1104-1117, 2020.

[0346]

[18] J. Ho, X. Chen, A. Srinivas, Y. Duan, and P. Abbeel, “Flow++: Improving flow-based generative models with variational dequantization and architecture design,” in Proc. of Machine Learning Research, 2019, vol. 97, pp. 2722-2730.

[0347]

[19] R. Prenger, R. Valle, and B. Catanzaro, “Waveglow: A Flow-based Generative Network for Speech Synthesis,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 3617-3621.

[0348]

[20] I. Kobyzev, S. Prince, and M. Brubaker, “Normalizing flows: An introduction and review of current methods,” IEEE Trans-actions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020.

[0349]

[21] G. Papamakarios, E.T. Nalisnick, D.J. Rezende, S. Mohamed, and B Lakshminarayanan, “Normalizing flows for probabilistic modeling and inference,” in arXiv:1912.02762, 2019.

[0350]

[22] D.P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” in Advances in Neural Information Processing Systems 31, 2018, pp. 10215-10224.

[0351]

[23] L Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real NVP,” in 5th Int. Conf. on Learning Representations, ICLR, 2017.

[0352]

[24] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1800-1807.

[0353]

[25] B. Zhai, T. Gao, F. Xue, D. Rothchild, B. Wu, J. Gonzalez, and K. Keutzer, “Squeezewave: Extremely lightweight vocoders for on-device speech synthesis,” in arXiv:2001.05685, 2020.

[0354]

[26] D. Rethage, J. Pons, and X. Serra, “A wavenet for speech de-noising,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5069-5073.

[0355]

[27] J. Jensen, I. Batina, R. C Hendriks, and R. Heusdens, “A study of the distribution of time-domain speech samples and discrete fourier coefficients,” in Proc. SPS-DARTS, 2005, vol. 1, pp. 155-158.

[0356]

[28] C. Valentini Botinhao, X. Wang, S. Takaki, and J. Yamagishi, “Speech enhancement for a noise-robust text-to-speech synthesis system using deep recurrent neural networks,” in Proc. In-terspeech Conf., 2016, pp. 352-356.

[0357]

[29] C. Veaux, J. Yamagishi, and S. King, “The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,” in Int. Conf. Oriental COCOSDA heldjointly with the Conf. on Asian Spoken Language Research and Evaluation (O-COCOSDA / CASLRE), 2013, pp. 1-4.

[0358]

[30] J. Thiemann, N. Ito, and E. Vincent, “The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings,” Proc. of Meetings on Acoustics, vol. 19, no. 1, pp. 035081, 2013.

[0359]

[31] D.P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd Int. Conf. on Learning Representations, ICLR, 2015.

[0360]

[32] T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural net-works,” in Advances in Neural Information Processing Systems 29, 2016, pp. 901-909.

[0361]

[33] M. Pariente, A. Deleforge, and E. Vincent, “A statistically principled and computationally efficient approach to speech enhancement using variational autoencoders,” in Proc. Inter-speech Conf., 2019, pp. 3158-3162.

[0362]

[34] Y. Hu and P. Loizou, “Evaluation of objective quality measures for speech enhancement,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, pp. 229 - 238, 02 2008.

[0363]

[35] J. Hansen and B. Pellom, “An effective quality evaluation protocol for speech enhancement algorithms,” in ICSLP, 1998.

[0364]

[36] R. Giri, U. Isik, and A. A. Krishnaswamy, “Attention wave-u-net for speech enhancement,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2019, pp. 249-253.

[0365]

[37] Y. Koizumi, K. Yatabe, M. Delcroix, Y. Masuyama, and D. Takeuchi, “Speech enhancement using self-adaptation and multi-head self-attention,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 181- 185.

Claims

1. A processed audio signal (y, 130, 230, 430) based on the input audio signal (y, 130, 230, 430) 1. An apparatus (100, 200, 400) for providing a liquid crystal display (CCD) The device (100, 200, 400) receives the processed audio signal ( , 160, 260, 460) to obtain one or more flow blocks (110 １．．．Ｎ , 210 １．．．Ｎ , 410 １．．．Ｎ ) for processing a noise signal (z, 120, 220, 420) or a signal derived from said noise signal (z, 120, 220, 420), The apparatus (100, 200, 400) generates the one or more flow blocks (110) in response to the input audio signal (y, 130, 230, 430) and using a neural network. １．．．Ｎ , 210 １．．．Ｎ , 410 １．．．Ｎ ) adapted to adapt processing performed using the

2. The apparatus (100, 200, 400) of claim 1, wherein the input audio signal (y, 130, 230, 430) is represented by a set of time-domain audio samples.

3. The one or more flow blocks (110 １．．．Ｎ , 210 １．．．Ｎ , 410 １．．．Ｎ 3. The apparatus (100, 200, 400) of claim 1 or 2, wherein a neural network associated with a given flow block of the noise signal (z, 120, 220, 420) is configured to determine one or more processing parameters of the given flow block in response to the noise signal (z, 120, 220, 420) or a signal derived from the noise signal (z, 120, 220, 420) and in response to the input audio signal (y, 130, 230, 430).

4. 4. The apparatus (100, 200, 400) of claim 1, wherein a neural network associated with a given flow block is configured to provide one or more parameters of an affine processing, the parameters being applied to the noise signal (z, 120, 220, 420), or a processed version of the noise signal, or a portion of the noise signal (z, 120, 220, 420), or a portion of the processed version of the noise signal during the processing.

5. A neural network associated with the given flow block calculates a first portion (z) of a flow block input signal (z). １ ) and the input audio signal (y, 130, 230, 430), and configured to determine one or more parameters of the affine processing in response to the input audio signal (y, 130, 230, 430); The affine processing associated with the given flow block is ) by applying the determined parameters to a second portion (z) of the flow block input signal (z) to obtain ２ ) The first portion (z) of the flow block input signal (z) １ ) and the affine processed signal ( ) is the flow block output signal (z ｎｅｗ 5. The apparatus (100, 200, 400) of claim 4, further comprising:

6. The apparatus (100, 200, 400) of claim 5, wherein the neural network associated with the given flow block includes a depthwise separable convolution in the affine process associated with the given flow block.

7. The apparatus (100, 200, 400) outputs a processed flow block output signal (z' ｎｅｗ ) to obtain the flow block output signal (z ｎｅｗ 7. The apparatus (100, 200, 400) of claim 5 or 6, configured to apply a reversible convolution to

8. The apparatus (100, 200, 400) of any one of claims 1 to 7, wherein the apparatus (100, 200, 400) is configured to apply a non-linear compression (490) to the input audio signal (y, 130, 230, 430) before processing the noise signal (z, 120, 220, 420) in response to the input audio signal (y, 130, 230, 430).

9. The apparatus (100, 200, 400) of claim 8, wherein the apparatus (100, 200, 400) is configured to apply a μ-law transform to the input audio signal (y, 130, 230, 430) as the nonlinear compression (490).

10. The device (100, 200, 400), and adapted to apply a transformation to said input audio signal (y, 130, 230, 430) according to sgn() is the sign function, 10. The apparatus (100, 200, 400) according to claim 8 or 9, wherein μ is a parameter defining a compression level.

11. The device (100, 200, 400) receives the processed audio signal ( 11. The apparatus (100, 200, 400) of claim 1, configured to apply a non-linear expansion (415) to a signal (signal) of the first order.

12. The device (100, 200, 400) applies an inverse μ-law transform to the processed audio signal ( 12. The apparatus (100, 200, 400) according to claim 11, configured for application to a method for detecting a temperature difference between a temperature range of the liquid crystal display and a temperature range of the liquid crystal display.

13. The device (100, 200, 400), According to the above, the processed audio signal ( , 160, 260, 460), sgn() is the sign function, 13. The apparatus (100, 200, 400) according to claim 11 or 12, wherein μ is a parameter defining an expansion level.

14. Neural network parameters of the neural network for processing the noise signal (z, 120, 220, 420) or the signal derived from the noise signal are a training result audio signal is obtained using processing of a training audio signal or a processed version thereof in one or more training flow blocks, the processing of the training audio signal or the processed version thereof using the one or more training flow blocks being adapted in response to a distorted version of the training audio signal and using the neural network; 14. The apparatus (100, 200, 400) of any one of claims 1 to 13, wherein the neural network parameters of the neural network are determined such that characteristics of the training result audio signal approximate or include predefined characteristics.

15. the device (100, 200, 400) is configured to provide neural network parameters of the neural network for processing the noise signal or the signal derived from the noise signal, the apparatus (100, 200, 400) is configured to process a training audio signal or a processed version thereof using the one or more flow blocks to obtain a training result audio signal; the apparatus (100, 200, 400) is configured to adapt, in response to a distorted version of the training audio signal and using the neural network, the processing of the training audio signal or a processed version thereof, which is performed using the one or more flow blocks, 15. The apparatus (100, 200, 400) of claim 1, wherein the apparatus (100, 200, 400) is configured to determine neural network parameters of the neural network such that characteristics of the training result audio signal approximate or include predetermined characteristics.

16. the apparatus comprising: an apparatus for providing neural network parameters; the device (100, 200, 400) for providing neural network parameters is configured to provide neural network parameters of the neural network for processing the noise signal or the signal derived from the noise signal, the apparatus (100, 200, 400) for providing neural network parameters is configured to process a training audio signal or a processed version thereof using one or more training flow blocks to obtain a training result audio signal, the device (100, 200, 400) for providing neural network parameters is configured to adapt, using the neural network, a processing of the training audio signal or a processed version thereof, which processing is performed using the one or more flow blocks in response to a distorted version of the training audio signal, The apparatus (100, 200, 400) of any one of claims 1 to 15, wherein the apparatus (100, 200, 400) is configured to determine neural network parameters of the neural network such that characteristics of the training result audio signal approximate or include predetermined characteristics.

17. The one or more flow blocks (110 １．．．Ｎ , 210 １．．．Ｎ , 410 １．．．Ｎ ) based on the noise signal (z, 120, 220, 420) under the guidance of the input audio signal (y, 130, 230, 430) 17. The apparatus (100, 200, 400) of claim 1, configured to synthesize a plurality of signals (a first signal, a second signal, a third signal, a fourth signal, a fifth signal, a sixth signal, a sixth signal, a sixth signal, a seventh ...

18. The one or more flow blocks (110 １．．．Ｎ , 210 １．．．Ｎ , 410 １．．．Ｎ ) based on the noise signal (z, 120, 220, 420) under the guidance of the input audio signal (y, 130, 230, 430) using affine processing of sample values ​​of the noise signal (z, 120, 220, 420) or a signal derived from the noise signal, , 160, 260, 460), The apparatus (100, 200, 400) of any one of claims 1 to 17, wherein processing parameters of the affine processing are determined based on sample values ​​of the input audio signal (y, 130, 230, 430) using the neural network.

19. The apparatus (100, 200, 400) of any one of claims 1 to 18, wherein the apparatus (100, 200, 400) is configured to perform a normalization flow process to derive the processed audio signal from the noise signal (z, 120, 220, 420).

20. 1. A method for providing a processed audio signal based on an input audio signal (y, 130, 230, 430), comprising the steps of: The method further comprises the steps of: , 160, 260, 460) to obtain one or more flow blocks (110 １．．．Ｎ , 210 １．．．Ｎ , 410 １．．．Ｎ ) to process a noise signal (z, 120, 220, 420), or a signal derived from said noise signal; The method further comprises: generating a plurality of flow blocks (y, 130, 230, 430) in response to the input audio signal (y, 130, 230, 430) using a neural network; , 160, 260, 460) to adapt the processing performed.

21. An apparatus (500, 600, 800) for providing neural network parameters for audio processing, comprising: The apparatus (500, 600, 800) comprises one or more flow blocks (510) for obtaining a training result audio signal (z, 520, 620, 820). １．．．Ｎ , 610 １．．．Ｎ , 810 １．．．Ｎ ) to process the training audio signal (x, 505, 605, 805) or a processed version thereof; The device (500, 600, 800) responsively performs a distorted version (y, 530, 630, 830) of the training audio signal and using a neural network to generate the one or more flow blocks (510 １．．．Ｎ , 610 １．．．Ｎ , 810 １．．．Ｎ ) adapted to adapt processing performed using The apparatus (500, 600, 800) is configured to determine neural network parameters of the neural network such that characteristics of the training result audio signal (z, 520, 620, 820) approximate or include predetermined characteristics.

22. the device (500, 600, 800) is configured to evaluate a cost function depending on characteristics of the obtained training result audio signal (z, 520, 620, 820), the apparatus (500, 600, 800) is configured to determine neural network parameters to reduce or minimize a cost defined by the cost function; 22. Apparatus (500, 600, 800) according to claim 21.

23. 23. The apparatus (500, 600, 800) of claim 21 or 22, wherein the training audio signal (x, 505, 605, 805) and / or the distorted version (y, 530, 630, 830) of the training audio signal is represented by a set of time-domain audio samples.

24. The one or more flow blocks (510 １．．．Ｎ , 610 １．．．Ｎ , 810 １．．．Ｎ 24. The apparatus (500, 600, 800) of claim 21, wherein a neural network associated with a given flow block of the audio signal train (x, 505, 605, 805) is configured to determine one or more processing parameters for the given flow block in response to the training audio signal (x, 505, 605, 805), or a signal derived from the training audio signal, and in response to a distorted version of the training audio signal (y, 530, 630, 830).

25. 25. The apparatus (500, 600, 800) of any one of claims 21 to 24, wherein a neural network associated with a given flow block is configured to provide one or more parameters of an affine process applied to the training audio signal (x, 505, 605, 805), or a processed version of the training audio signal, or a portion of the training audio signal (x, 505, 605, 805), or a portion of the processed version of the training audio signal being processed.

26. A neural network associated with the given flow block calculates a first portion (x) of a flow block input signal (x). １ ) or in response to a first portion of a pre-processed flow block input signal (x′) and in response to a distorted version of the training audio signal (y, 530, 630, 830), The affine processing associated with the given flow block is ) by applying the determined parameters to a second portion (x) of the flow block input signal (x) to obtain ２ ) or a second portion of the pre-processed flow block input signal (x′), The first portion (x) of the flow block input signal (x) or the pre-processed flow block input signal (x′) １ ) and the affine processed signal ( ) is the flow block output signal x of the given flow block ｎｅｗ 26. The apparatus (500, 600, 800) of claim 25,

27. 27. The apparatus (100, 200, 400) of claim 26, wherein the neural network associated with the given flow block includes a depthwise separable convolution in the affine process associated with the given flow block.

28. The apparatus (500, 600, 800) of claim 26 or 27, wherein the apparatus (500, 600, 800) is configured to apply a reversible convolution to the flow block input signal (x) of the given flow block to obtain the pre-processed flow block input signal (x').

29. 29. The apparatus (500, 600, 800) of any one of claims 21 to 28, wherein the apparatus (500, 600, 800) is configured to apply a non-linear input compression (815) to the training audio signal (x, 505, 605, 805) before processing the training audio signal (x, 505, 605, 805).

30. 30. The apparatus (500, 600, 800) of claim 29, wherein the apparatus (500, 600, 800) is configured to apply a μ-law transform to the training audio signal (x, 505, 605, 805) as the nonlinear input compression (815).

31. The device (500, 600, 800), and adapted to apply a transformation to said training audio signal (x, 505, 605, 805) according to sgn() is the sign function, (500, 600, 800) according to claim 29 or 30, wherein μ is a parameter defining the compression level.

32. The apparatus (500, 600, 800) of any one of claims 21 to 31, wherein the apparatus (500, 600, 800) is configured to apply a nonlinear input compression (815) to the distorted version (y, 530, 630, 830) of the training audio signal before processing the training audio signal (x, 505, 605, 805) in response to the distorted version (y, 530, 630, 830) of the training audio signal.

33. 33. The apparatus (500, 600, 800) of claim 32, wherein the apparatus (500, 600, 800) is configured to apply a μ-law transform to a distorted version (y, 530, 630, 830) of the training audio signal as the nonlinear input compression (815).

34. The device (500, 600, 800), and adapted to apply a transformation to a distorted version (y, 530, 630, 830) of the training audio signal according to sgn() is the sign function, 34. Apparatus (500, 600, 800) according to claim 32 or 33, wherein μ is a parameter defining a compression level.

35. The one or more flow blocks (510 １．．．Ｎ , 610 １．．．Ｎ , 810 １．．．Ｎ 35. The apparatus (500, 600, 800) of any one of claims 21 to 34, wherein a training audio signal (x, 505, 605, 805) is configured to convert the training audio signal (x, 505, 605, 805) into the training result audio signal (z, 520, 620, 820).

36. The one or more flow blocks (510 １．．．Ｎ , 610 １．．．Ｎ , 810 １．．．Ｎ ) is adapted to transform said training audio signal (x, 505, 605, 805) into said training result audio signal (z, 520, 620, 820) under the guidance of a distorted version (y, 530, 630, 830) of said training audio signal using affine processing of sample values ​​of said training audio signal (x, 505, 605, 805) or of a signal derived from said training audio signal (x, 505, 605, 805), 36. The apparatus (500, 600, 800) of any one of claims 21 to 35, wherein processing parameters of the affine processing are determined based on sample values ​​of a distorted version (y, 530, 630, 830) of the training audio signal using the neural network.

37. 37. The apparatus (500, 600, 800) of any one of claims 21 to 36, wherein the apparatus (500, 600, 800) is configured to perform a normalization flow process to derive the training result audio signal (z, 520, 620, 820) from the training audio signal (x, 505, 605, 805).

38. 1. A method for providing neural network parameters for audio processing, comprising: The method further comprises the steps of: obtaining a training result audio signal (z, 520, 620, 820); １．．．Ｎ , 610 １．．．Ｎ , 810 １．．．Ｎ ) to process the training audio signal (x, 505, 605, 805) or a processed version thereof; The method further comprises: generating a training signal from the one or more flow blocks (510, 630, 830) in response to the distorted version of the training audio signal (y, 530, 630, 830) and using a neural network. １．．．Ｎ , 610 １．．．Ｎ , 810 １．．．Ｎ ) adapting said processing to be performed using 4. A method according to claim 3, further comprising: determining the neural network parameters of the neural network such that characteristics of the training result audio signal (z, 520, 620, 820) approximate or include predetermined characteristics.

39. 39. A computer program having a program code for performing the method according to any one of claims 20 or 38, when the computer program runs on a computer.

Images