Wiener-filter-based signal restoration with learned signal-to-noise ratio estimate

Abstract

The disclosure relates to a method for Wiener-filter-based signal restoration, comprising the following method steps: receiving a signal (g); estimating a signal-to-noise ratio for a Wiener-filter-based restoration algorithm (v) by a processing algorithm (φ) obtained by means of a machine learning processing, depending on a spectral power density calculated for the received signal; and generating a restored signal (ŝ) from the received signal (g) and from the signal-to-noise ratio estimated for the Wiener-filter-based restoration algorithm (v) by means of the Wiener-filter-based restoration algorithm (v) in order to improve the filter-based signal restoration, in particular the result of a Wiener-filter-based signal restoration.

Description

[0001] Fraunhofer Society 239PCT 1678 GD

[0002] Wiener filter-based signal restoration with learned signal-to-noise ratio estimation

[0003] The present disclosure relates to a method and apparatus for Wiener filter-based signal restoration, in which a signal is received, a signal-to-noise ratio of the signal is estimated for use in a Wiener filter-based restoration algorithm, and then, using the Wiener filter-based restoration algorithm, an original signal is restored from the received signal, taking into account the estimated signal-to-noise ratio, ie, a (restored) signal that is as similar as possible to the original signal. Generally, signals transmitted on signal or reception paths are degraded, ie,An original signal is corrupted, on the one hand, by non-ideal transmission to a corresponding receiving sensor, mathematically represented by a non-ideal mapping function, and on the other hand, by external interference, mathematically represented by an interference signal. As a result, the observed or received signal always deviates from the original signal. Typically, a restoration filter function is therefore applied to the observed signal, generating a restored signal. The restored signal is an estimate of the original signal, as various assumptions must be made when selecting the restoration function, thus preventing perfect restoration. As a restored signal, it is considered equivalent to the original signal for further use.

[0004] For example, if images are captured with a camera system, physically induced image degradation may occur depending on the situation. Some image degradations can be formulated as linear, shift-invariant systems and thus fully described based on their impulse response. Examples of this are blurred images, image defects caused by suboptimal optics, motion blur, and the like. From a systems-theoretical perspective, the captured image as the observed signal then corresponds to a convolution of the undisturbed image, the original signal, with the impulse response of the existing image degradation, the non-ideal imaging function. In such cases, depending on the severity of the image degradation and the presence of image noise as an additional interference signal, it is possible to a certain extent to calculate an image as a restored signal using image reconstruction or restoration methods that very closely approximates the original image.In theory, this task can be optimally solved using the so-called Wiener filter. In practice, however, the Wiener filter has the critical disadvantage that the signal-to-noise ratio required for filtering with the Wiener filter is unknown and can essentially only be estimated. As a result, the filter result of the Wiener filter is generally unsatisfactory and is generally post-processed to achieve a better result. In the article "A Data Driven Approach to A Priori SNR Estimation" by Suhadi S. et al., published in 2011 in IEEE Transactions on Audio, Speech, and Language Processing 19, pages 186 to 195, the Wiener filter is used to enhance the signal in speech processing. Two convolutional neural networks are trained that can detect regions with and without speech in the temporal signal.Assuming that the noise is similar in both regions, the signal-to-noise ratio (SNR) can be estimated by calculating the corresponding signal components. However, this approach is not applicable to image signals, for example, because the Wiener filter is described in the spatial frequency domain and not with respect to individual pixels or image regions.

[0005] In the article "An Iterative SNR Estimation Algorithm for Wiener Deconvolution of Self-Similar Images Distorted by Camera Shake Blurring" by Marcelo AP et al., published in 2008 in the Proceedings of the 8th Conference on Signal, Speech and Image Processing on pages 97 to 100, an initial SNR estimate is first used to restore the input image with the Wiener filter. The resulting image, as the restored image, is compared with the input image in terms of the similarity of the gradients in the x and y directions, in order to then adjust the SNR accordingly. This is followed by the next iteration.

[0006] In the article "SNR-Aware Convolutional Neural Network Modeling for Speech Enhancement" by Fu S.-W. et al., published in 2016 in Interspeech on pages 3268 to 3772, a speech signal is processed by a convolutional neural network to estimate the SNR for each time period considered. However, only an average SNR value is estimated here, rather than separate SNR values ​​for all available frequencies, as required for the Wiener filter.

[0007] The task is therefore to improve filter-based signal restoration, in particular the result of Wiener filter-based signal restoration.

[0008] This problem is solved by the subject matter of the independent patent claims. Advantageous embodiments emerge from the dependent patent claims, the description, and the figures.

[0009] The approach presented below is based on the usual signal model for signal restoration, as is known, for example, from image restoration. An original signal is transformed by a non-ideal mapping function h. In addition, the transformed signal is corrupted by a disturbance n, thus yielding the observed or received signal g. Applying a restoration function v to the observed or received signal yields a restored signal s. The signals s, g, s as well as functions h and v and the disturbance n can, as is typically the case with image signals, exhibit a dependence on a location x; in other application areas, for example, they can also exhibit a dependence on a frequency f and the like. Using the nomenclature presented, the Wiener filter for the case of an image signal in the frequency domain is obtained according to

[0010] Here, / 7(f) = T{h(x)} describes the transfer function of the image degradation, i.e., the Fourier transform of the impulse response as a non-ideal mapping function h(x). To use the Wiener filter, the expression SNR(f) = S s s(f) / S n n(f ) should be determined or estimated as accurately as possible. Here, S ss (f) the unknown and thus to be estimated spectral power density of the undisturbed original signal s and S nn (f) the unknown and therefore to be estimated spectral power density of, for example, noise as disturbance n.

[0011] One aspect of the presented approach accordingly concerns a method for Wiener filter-based signal restoration, also referred to as data signal restoration, comprising the method steps of receiving a signal, the observed signal g, estimating the signal-to-noise ratio for restoring the original signal s underlying the received signal g in the form of a restored signal s, and generating the restored signal s from the received signal g and the estimated SNR. The method steps are carried out by a signal processing unit, which may, for example, contain a microprocessor and corresponding further electronic elements.The signal belongs to a respective signal type, for example, it can be an image signal, in particular a single- or multi-channel image signal, and / or an audio signal, and / or a digital data transmission signal, or the signal can each comprise one or more signals of the corresponding signal type "image signal" and / or "audio signal" and / or "data transmission signal". Accordingly, the received signal can be generated and / or received by an image sensor unit and / or audio sensor unit and / or a data transmission unit. The signal is received on a respective reception path, wherein the received or observed signal is formed by a corruption of the original signal by or on the reception path.The distortion can be caused by the nature of the reception path itself, which is then described by the non-ideal mapping function h, or by additional disturbances which are described by the disturbance factor n.

[0012] The estimation of the signal-to-noise ratio for a Wiener filter-based restoration algorithm is performed by a processing algorithm obtained by means of a machine learning method. The processing algorithm obtained by means of the machine learning method can be or comprise a neural network, in particular a deep neural network with two or more, preferably three or more, hidden layers. However, other machine learning methods such as pixel-wise support vector regression can also be used. The estimation is performed as a function of, i.e., a spectral power density S calculated for the received signal. gg .

[0013] The restored signal s is generated from the received, i.e. observed signal g and the signal-to-noise ratio SNR estimated for the Wiener filter-based restoration algorithm v using the Wiener filter-based restoration algorithm v. The signal-to-noise ratio SNR estimated by the processing algorithm obtained in machine learning methods forms the basis of the Wiener filter of the Wiener filter-based restoration algorithm v. In contrast to known methods in which a result of a Wiener filter-based restoration algorithm is subsequently optimized, the method presented here directly addresses the weakness of the Wiener filter, namely the signal-to-noise ratio which is often difficult to estimate correctly in practice.As a result, the theoretical optimality of the Wiener filter also comes into full effect in practical applications - various experiments have shown that the approach presented here typically achieves the restoration of signals with a quality that exceeds the performance of known approaches in common quality metrics by 10%, i.e. 10 percentage points.

[0014] Accordingly, in an advantageous embodiment, the method also comprises training the processing algorithm obtained by means of the machine learning method with a plurality of training signal-data pairs. These training signal-data pairs each comprise or contain a spectral power density calculated for a received training signal of the same signal type as the signal subsequently received in the application and a training signal-to-noise ratio calculated as a function of an original training signal and a predetermined noise training signal. The training described here and below can also be carried out independently of the signal restoration itself, i.e., spatially and / or temporally separated from the actual Wiener filter-based signal restoration.This has the advantage that the processing algorithm developed using the machine learning method can quickly estimate the SNR in practice, since only the observed signal is required to estimate the respective SNR. Since very large existing databases of signals such as images, audio signals, and other signals, as well as corresponding non-ideal mapping functions such as impulse responses of receive paths, can be used for training, this type of training is also suitable for practical use.

[0015] In an advantageous embodiment, it is provided that the spectral power density calculated for the received signal during the estimation is a logarithmic power density, i.e. the calculated spectral power density is logarithmized after the calculation and before further processing, and the spectral power density calculated for the received training signal during training is correspondingly a logarithmic power density, just as the training signal-to-noise ratio calculated as a function of the original training signal and the predetermined noise training signal is a logarithmic training signal-to-noise ratio, i.e. the SNR is also logarithmized after the calculation before further processing.Before restoring the original signal, the signal-to-noise ratio estimated for the Wiener filter-based restoration algorithm is then exposed to compensate for any distortions induced by the logarithmization of the input variable. This has the advantage that the machine learning method converges better, especially when it is a neural network, especially a deep neural network. Especially with image data, estimating the spectral power density using the advantageous squared magnitude of the discrete Fourier transform impairs the convergence behavior of the aforementioned machine learning methods.

[0016] In a further advantageous embodiment, the respective received training signal is calculated as a function of the respective associated original training signal, i.e., the original training signal of the same pair, and a respective impulse response training signal. Thus, with access to the different databases, the amount of training data can be significantly increased again, thus increasing the performance of the processing algorithm. In addition, the respective received training signal can also depend on the specified noise training signal.

[0017] In another advantageous embodiment, the (non-logarithmic) training signal-to-noise ratio calculated as a function of the original training signal and the specified noise training signal comprises the quotient of the spectral power density calculated for the original training signal and the spectral power density calculated for the specified noise training signal, in particular, is proportional to this quotient or is the quotient. The SNR is thus estimated or calculated using the quotient or as the quotient of the respective spectral power densities. This leads to good restoration results, especially in combination with the method of calculating the received training signal with the associated spectral power density described in the last paragraph.

[0018] A further aspect relates to a signal processing unit for Wiener filter-based signal restoration, which is designed to carry out a method according to one of the described embodiments, i.e. the Wiener filter-based signal restoration and / or the training of the processing algorithm obtained by means of machine learning methods as described for this purpose.

[0019] Advantages and advantageous embodiments of the signal processing unit correspond to advantages and advantageous embodiments of the respective methods.

[0020] The features and feature combinations mentioned above in the description, including in the introductory part, as well as the features and feature combinations mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the respective combination specified, but also in other combinations without departing from the scope of the invention. Thus, embodiments are also to be considered encompassed and disclosed by the invention that are not explicitly shown and explained in the figures, but which emerge and can be produced by separate feature combinations from the explained embodiments. Embodiments and feature combinations are also to be considered disclosed that therefore do not have all the features of an originally formulated independent claim.Furthermore, embodiments and combinations of features are to be regarded as disclosed, in particular by the embodiments set out above, which go beyond or deviate from the combinations of features set out in the reliances of the claims.

[0021] Fig. 1 shows a signal path for a reception path with subsequent restoration according to a known signal model; and

[0022] Fig. 2 shows a schematic overview of an exemplary training procedure for a processing algorithm obtained by means of machine learning.

[0023] In the figures, identical or functionally identical elements are provided with the same reference symbols.

[0024] Fig. 1 shows a generally known signal model for signal restoration. An original signal s is shaped on the receive path by its specific properties, which are modeled by a non-ideal mapping function h, which is applied to the original signal s, for example by convolution. The signal is additionally additively corrupted by an external disturbance n, resulting in a signal g, which is then observed or received. This observed or received signal g is transformed by a restoration filter function v, which can also be referred to as a restoration algorithm v, so that the restoration result is a restored signal s, an estimate of the original or original signal s. With image signals as exemplary signals, and thus signals dependent on a location x orFunctions s, h, n, g, v, s result in the Wiener filter in the frequency domain using the formula already presented:.

[0025] The decisive factor for the quality of the restoration result is the most accurate determination of the signal-to-noise ratio SNR = S s s / s n n, where in this example the respective terms SNR, S ss and S nn are linked to the frequency f via a Fourier transformation by the vector x.

[0026] Fig. 2 schematically illustrates an exemplary embodiment of a method for training a processing algorithm obtained by machine learning, here a neural network 4>. The neural network 4> is trained to estimate the desired SNR based on an estimate of the spectral power density S^ of the received signal g, for example, of an observed image g(x), in the case of a location-dependent image g(x), the desired SNR is SNR (f). Thus, 4>(^) = SNR. Here, S gg (f) = |T'(s r (x))| 2 with the Fourier transform T{.} and the estimation of the signal-to-noise ratio SNR. In Fig. 2, the Fourier transform T{.} is chosen as an example of a discrete Fourier transform DFT {.}.

[0027] In the example shown, an original signal s, in this case an image s(x), is selected from a first database D1 to train the neural network 4>. From a second database D2, which can be any signal degradation database, a corresponding impulse response is selected as a non-linear mapping function h, here h(x). The original signal s is convolved with the impulse response as a non-ideal mapping function h in order to simulate the signal degradation during training. For image data, for example, the image degradation databases from the article "Understanding and Evaluating Blind Deconvolution Algorithms" by Levin A. et al., published in 2009 in the IEEE Conference on Computervision and Pattern Recognition on pages 1964 to 1971, or from the article "Edge-Based Blur Kernel Estimation using Patch Priors" by Libin Sun et al., published in 2013 in the IEEE International Conference on Computational Photography on pages 1 to 8. The disturbance n, simulated, for example, as normally distributed noise n(x), is added to the convolution result. The result is a simulated received signal g, here g(x). The logarithm of the squared magnitude of the discrete Fourier transform (DFT) is calculated from this simulated received signal g, which is the input signal log S. gg for the neural network 4>.

[0028] In addition, the logarithmic quotient log the logarithmized signal-to-noise ratio log SNR, which is later estimated during signal recovery by the processing algorithm, here the neural network 4>, is calculated, which determines or forms a reference input for the training of the neural network 4>.

[0029] Using the logarithms log S^ and log SNR to train the neural network instead of S^ and SNR serves to reduce the dynamic range of the resulting values. Accordingly, after evaluating 4>, the value output by the neural network 4> must be exponentiated, and the desired signal-to-noise ratio (SNR) obtained by the processing algorithm is then given by

[0030] The transfer function h required for reconstruction, or its Fourier transform H of the signal degradation, can be calculated using other existing methods. If the signals are image data, for example, motion blur can be estimated using data from an acceleration sensor or gyroscope of the recording device, such as a smartphone.

Claims

Claims 1. Method for Wiener filter-based signal restoration, with the following steps: - receiving a signal (g); - estimating a signal-to-noise ratio for a Wiener filter-based restoration algorithm (v) by a processing algorithm (4>) obtained by means of a machine learning method, as a function of a power spectral density calculated for the received signal; - generating a restored signal (s) from the received signal (g) and the signal-to-noise ratio estimated for the Wiener filter-based restoration algorithm (v) by means of the Wiener filter-based restoration algorithm (v).

2. Method according to claim 1, characterized in that the signal is or comprises an image signal and / or an audio signal and / or a digital data transmission signal.

3. Method according to one of the preceding claims, characterized in that the signal is generated by an image sensor unit and / or by an audio sensor unit and / or a data transmission unit.

4. Method according to one of the preceding claims, characterized in that the processing algorithm (4>) obtained by means of the machine learning method comprises a neural network, in particular a deep neural network.

5. Method according to one of the preceding claims, characterized by a - Training the processing algorithm (4>) obtained by means of the machine learning method with a plurality of training signal-data pairs, each of which comprises a spectral power density calculated for a received training signal (g) and a training signal-to-noise ratio calculated as a function of an original training signal (s) and a predetermined noise training signal (n).

6. Method according to claim 5, characterized in that - the spectral power density calculated in the estimation for the received signal (g) is a logarithmic power density, and - the spectral power density calculated during training for the received training signal (g) is a logarithmic power density and the training signal-to-noise ratio calculated as a function of the original training signal (s) and the predetermined noise training signal (s) is a logarithmic training signal-to-noise ratio, where - before generating the recovered signal (s), the signal-to-noise ratio estimated for the Wiener filter-based recovery algorithm (v) is exposed.

7. Method according to claim 5 or 6, characterized in that the respective received training signal (g) is calculated as a function of the respectively associated original training signal (s) and a respective impulse response training signal (h).

8. The method according to claim 5 or 6 or 7, characterized in that the training signal-to-noise ratio calculated as a function of the original training signal (s) and the predetermined noise training signal (n) comprises the quotient of the spectral power density calculated for the original training signal (s) with the spectral power density calculated for the predetermined noise training signal (n), in particular is proportional to this or is the quotient.

9. Method for training the processing algorithm (4>) obtained by machine learning for a Wiener filter-based recovery algorithm (v) according to one of the preceding claims.

10. A signal processing unit for Wiener filter-based signal restoration, which is designed to carry out a method according to one of the preceding claims.

Images