Robustness / performance improvement for deep learning based speech enhancement against artifact and distortion

Abstract

To provide a method for processing audio signals, as well as a corresponding device, a computer program, and computer-readable storage media.SOLUTION: A method 1000 includes the steps of: applying emphasis to a first component of an audio signal and / or applying suppression to a second component of the audio signal with respect to the first component S1010; and modifying an output of the step S1010 by applying a deep learning based model to the output to remove artifacts and / or distortions introduced into the audio signal in the step S1010, and perceptually improving the first component of the audio signal S1020.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to the field of audio processing. In particular, the present disclosure relates to techniques for audio enhancement (e.g., audio enhancement) using deep-learning models or systems, and to a framework for training deep-learning models or systems for audio enhancement.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to International Application No. PCT / CN2021 / 082199, filed March 22, 2021, European Patent Application No. 21178178.6, filed June 8, 2021, and U.S. Provisional Patent Application No. 63 / 180,705, filed April 28, 2021, each of which is incorporated by reference in its entirety herein. [Background technology]

[0003] Speech enhancement aims to enhance or separate a speech signal (speech component) from a noisy mixture signal. Numerous speech enhancement methods have been developed over the past decades. Recently, speech enhancement has been formulated as a supervised learning task, where discriminative patterns of clean speech and background noise are learned from training data. However, these algorithms all suffer from different processing distortions when dealing with different acoustic environments. Typical processing distortions include target loss, interference, and algorithmic artifacts.

[0004] Thus, there is a need for improved deep learning-based methods of audio processing, including speech enhancement, that can reduce artifacts and / or distortion. Summary of the Invention

[0005] In view of the above, the present disclosure provides a method for processing an audio signal, as well as a corresponding device, a computer program and a computer-readable storage medium, as characterized in the respective claims.

[0006] According to one aspect of the disclosure, a method for processing an audio signal is provided. The method may include a first step of applying an emphasis to a first component of the audio signal and / or applying a suppression to a second component of the audio signal relative to the first component. The first step may be an emphasis or separation step, at least partially separating the first component from any residual components of the audio signal or generating a mask to do so. Thus, the first step may also be said to perform a denoising operation. The emphasis of the first component may be relative to the second component. The first component may be, for example, speech (speech component). The second component may be, for example, noise (noise component) or background (background component). The method may further include a second step of modifying the output of the first step by applying a deep learning-based model to the output of the first step. The second step removes artifacts and / or distortions introduced by the first step in the audio signal and perceptually improves the first component of the audio signal. The second step may be a modification step or a refinement step. It may result in the removal of distortions and / or artifacts introduced by the first step. The second step may operate on the waveform signal having the first component emphasized and / or the second component suppressed, or may operate on a mask, depending on the output of the first step.

[0007] Configured as described above, the proposed method is able to remove artifacts and distortions introduced by audio processing steps, such as speech enhancement steps (e.g., deep learning-based speech enhancement steps). This is achieved by means of a deep learning-based model that can be specifically trained to remove artifacts and distortions resulting from the audio processing at hand.

[0008] In some embodiments, the first step may be a step of applying speech enhancement to the audio signal, such that the first component may correspond to a speech component and the second component may correspond to a noise, background or residual component.

[0009] In some embodiments, the output of the first step is a waveform-domain audio signal (e.g. a waveform signal) in which the first component is emphasized and / or the second component is suppressed relative to the first component. Thus, the first step can receive a time-domain (waveform-domain) audio signal and apply the emphasis of the first component and / or the suppression of the second component by (directly) modifying the time-domain audio signal.

[0010] In some embodiments, the output of the first step may be a mask in the transform domain indicating weighting factors for individual bins or bands. The transform domain (transformed region) may be, for example, the frequency domain or the spectral domain. The (transform domain) bins may be time-frequency bins. The mask may be, for example, an amplitude mask, a phase-sensitive mask, a complex mask, a binary mask, etc. Furthermore, applying the mask to the (transform domain) audio signal may result in an emphasis of the first component and / or a suppression of the second component relative to the first component. In particular, the emphasis of the first component and / or the suppression of the second component may be achieved by applying a mask to the transform domain audio signal by removing or suppressing time-frequency tiles associated with noise or background. It is understood that the method may optionally include an (initial) step of transforming the audio signal into the transform domain and / or a (final) step of performing an inverse transform.

[0011] In some embodiments, the second step may receive a plurality of instances of the output of the first step, where each of the instances may correspond to a respective one of a plurality of frames of the audio signal. Furthermore, the second step applies a machine learning based model to the plurality of instances of the output together to perceptually refine the first component of the audio signal in one or more of the plurality of frames of the audio signal. In this case, the deep learning based model of the second step may have been trained based on the plurality of instances of the output of the first step and the corresponding plurality of frames of a reference audio signal for the audio signal. Alternatively, both the training and operation of the second step may proceed frame by frame.

[0012] In some embodiments, the second step may receive a sequence of instances of the output of the first step for a given frame of the audio signal, where each of the instances may correspond to a respective one in a sequence of frames of the audio signal. The sequence of frames may include the given frame (e.g. as its last frame). For example, the operation of the second step may be based on a shifting window of frames including the given frame. Thus, the method may maintain a history of previous frames (i.e. prior to the given frame) to be considered when generating an output for the given frame. Furthermore, the second step may apply the machine learning based model together to the sequence of instances of the output to perceptually improve the first component of the audio signal at the given frame.

[0013] In some embodiments, the deep learning-based model of the second step may implement an autoencoder architecture having an encoder stage and a decoder stage. Each stage may comprise a respective plurality of successive filter layers. The encoder stage may map the input to the encoder stage to a latent space representation (e.g., a code). The input to the encoder stage (i.e., the output of the first step) may be, for example, the mask described above. The decoder stage may map the latent space representation output by the encoder stage to an output of the decoder stage having the same format as the input to the encoder stage. The encoder stage may be said to successively reduce the dimensionality of the input to the encoder stage, and the decoder stage may be said to successively enhance the dimensionality of the input to the decoder stage back to the original dimensionality. Thus, the format of the input / output may correspond to the dimensionality of the input / output.

[0014] In some embodiments, the deep learning based model of the second step may implement a recurrent neural network architecture having multiple successive layers, where the multiple layers may be of long short-term memory type or gated recurrent unit type layers.

[0015] In some embodiments, the deep learning-based model may implement a generative model architecture having multiple consecutive convolutional layers, where the convolutional layers may be dilated convolutional layers, which may optionally include one or more skip connections between the convolutional layers.

[0016] In some embodiments, the method further comprises one or more additional first steps for applying emphasis to the first component of the audio signal and / or applying suppression to the second component of the audio signal, where the first step and the one or more additional first steps may generate mutually different outputs (e.g., pairwise different). Otherwise, the one or more additional first steps may have the same purpose or intent as the first step. In this configuration, the second step may receive an output of each of the one or more additional first steps in addition to the output of the first step. Furthermore, the second step applies the deep learning-based model jointly to the output of the first step and the output of the one or more additional first steps to perceptually improve the first component of the audio signal. The second step may, among others, apply weighting and / or selection to the outputs of the first step and the one or more additional first steps. It is possible.

[0017] In some embodiments, the method further comprises a third step of applying a deep learning based model to the audio signal to band the audio signal prior to input to the first step. The second step may then modify the output of the first step by debanding the output of the first step. The deep learning based models of the second and third steps may have been trained together.

[0018] In some embodiments, the second and third steps may implement multiple successive layers with successively increasing and decreasing numbers of nodes, respectively. That is, the second and third steps may implement an autoencoder architecture, with the third step corresponding to an encoder (encoder stage) and the second step corresponding to a decoder (decoder stage). The first step may operate on the code (latent space representation) generated by the third step.

[0019] In some embodiments, the first step may be a deep learning based model for enhancing the first component of the audio signal and / or suppressing the second component of the audio signal relative to the first component (the first step may apply a deep learning based model as a second step). For example, the first step may be a deep learning based speech enhancement step.

[0020] According to another aspect of the present disclosure, there is provided an apparatus for processing an audio signal, the apparatus may include a first stage for applying emphasis to a first component of the audio signal and / or suppression to a second component of the audio signal relative to the first component, and may further include a second stage for modifying an output of the first stage by applying a deep learning based model to an output of the first stage, removing artifacts and / or distortions introduced by the first stage into the audio signal and perceptually improving the first component of the audio signal.

[0021] According to another aspect, a computer program is provided which may include instructions that, when executed by a processor, cause the processor to perform all the steps of the methods described throughout the present disclosure.

[0022] According to another aspect, a computer readable storage medium is provided, which can store the computer program described above.

[0023] According to yet another aspect, an apparatus is provided that includes a processor and a memory coupled to the processor, the processor may be adapted to perform all steps of the methods described throughout this disclosure.

[0024] It will be understood that apparatus features and method steps can be interchanged in many ways. In particular, details of the disclosed methods can be implemented by corresponding apparatus, and vice versa, as will be understood by those skilled in the art. Furthermore, any statements made above with respect to the methods (and, e.g., steps thereof) will be understood to apply equally to the corresponding apparatus (and, e.g., blocks, stages, units thereof), and vice versa. [Brief description of the drawings]

[0025] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. [Figure 1] FIG. 1 illustrates a schematic diagram of an example of an apparatus (eg, a system or device) for implementing a method of audio processing according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 illustrates a schematic diagram of an example of an apparatus (eg, a system or device) for implementing a method of audio processing according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 illustrates a schematic diagram of an example of a processing block of an apparatus according to an embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates a further example of an apparatus for implementing the method of audio processing according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 illustrates a further example of an apparatus for implementing the method of audio processing according to an embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates a further example of an apparatus for implementing the method of audio processing according to an embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates a further example of an apparatus for implementing the method of audio processing according to an embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates generally an example of a framework for employing an apparatus implementing a method of audio processing according to an embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates a schematic diagram of yet another example of an apparatus for implementing a method of audio processing according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a flow chart that generally illustrates an example of a method for audio processing, in accordance with an embodiment of the present disclosure. [Figure 11] FIG. 11 is a flow chart that generally illustrates another example of a method of audio processing, in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The drawings and the following description relate to preferred embodiments by way of example only. It should be noted from the following description that alternative embodiments of the structures and methods disclosed herein are readily recognizable as viable alternatives that may be used without departing from the principles of what is claimed.

[0027] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that, wherever possible, like or similar reference numbers may be used in the figures and may indicate like or similar functionality. The drawings depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods described herein may be employed without departing from the principles described herein.

[0028] As mentioned above, conventional deep learning-based speech enhancement typically introduces distortion and artifacts. To alleviate this problem, the present disclosure proposes a multi-stage deep learning-based speech enhancement framework that can reduce artifacts and distortion. The framework includes two blocks, namely, a "separator" and an "improver". Here, the separator is used to perform a first round of denoising, and the subsequent improver helps to reduce distortion and remove artifacts introduced by the separator. In addition, the improver can also act as a "manager" that merges and balances the outputs of a set of separators to finally output a comprehensive result.

[0029] In particular, although this disclosure often refers to speech enhancement (e.g., in a first stage), it is understood that this disclosure relates generally to any audio processing or audio enhancement, both conventional and deep learning based, in a first stage that may introduce distortions and / or artifacts.

[0030] Methodology Overview Speech enhancement has recently been formulated as a supervised learning task, where discriminatory patterns of clean speech and background noise are learned from training data. Currently, supervised speech enhancement algorithms can be essentially categorized into two groups. One group includes wave domain-based models, and the other group includes transform domain-based models. The target of the waveform domain-based models is essentially a clean wave, while for the transform domain-based models, the target can be a bin-based mask (e.g., a magnitude mask, a phase-sensitive mask, a complex mask, a binary mask, etc.) or a band-based mask, depending on the respective use case. While some implementations of the present disclosure may be based on or relate to spectral domain processing, it is understood that the present disclosure is not so limited and relates to waveform domain processing as well.

[0031] Given a mixture y (e.g., an input audio signal), which can be a mono, stereo, or even a multi-channel signal, the goal of speech enhancement is to separate a target speech s (e.g., speech component) from a background n (e.g., background, noise, or residual component). The noise signal y can be modeled as follows:

number

number

number

[0032] Different speech enhancement algorithms have different kinds of distortions, which may also be correlated with noise types and signal-to-noise conditions. To derive a speech enhancer that is robust to processing artifacts, this disclosure proposes a new model framework that includes two blocks: a "separator" block and a "refiner" block.

[0033] FIG. 1 is a block diagram that illustrates a schematic of an apparatus (e.g., a system or device) according to this model framework. A system 100 for speech enhancement (or, in general, audio enhancement / processing) comprises a separator block (separator) 10 and an improver block (improver) 20. An audio signal 5 (e.g., containing the aforementioned mixture y) is input to the system 100. The separator 10 implements the first stage or step of the proposed model framework. It is used to perform a first round of denoising of the input audio signal 5. An output 15 of the separator 10 may relate to a modified version of the input audio signal 5 (i.e., the audio signal in the waveform domain) or to a mask that can be applied to the audio signal in the transform domain, as will be explained in more detail later.

[0034] The downstream improver 20 implements the second stage or step of the model framework. It receives and acts on the output 15 of the separator 10. The improver 20 processes the output 15 of the separator 10 to reduce the target distortion, remove or suppress artifacts, and / or remove or suppress residual noise in the audio signal. The improver 20 finally produces an output 25, which may relate to a (further) modified waveform signal or a modified mask, as will be explained in more detail below. It should be noted that the proposed framework does not relate to the concatenation of two separate models, but in fact to a single integrated model. The separator 10 and the improver 20 are just two (conceptual) blocks in the model.

[0035] In particular, the separator 10 may be implemented as a deep neural network (DNN) or by conventional audio processing components, while the improver 20 according to the present disclosure is implemented by a deep neural network, i.e., is deep learning based.

[0036] While there are many separators 10 proposed in academia and industry, this disclosure focuses primarily on the improvements 20, including potential structures and implementations, working with the separator 10, and training strategies.

[0037] In line with the above, one example of a method 1000 for audio processing (e.g., audio enhancement, such as speech enhancement) is shown generally in the flow chart of Fig. 10. The method 1000 may be, for example, a method for speech enhancement. It includes steps S1010 and S1020.

[0038] Step S1010 is a first step for applying enhancement to a first component (e.g. speech) of the audio signal 5 and / or suppression to a second component (e.g. noise or background) of the audio signal 5 (e.g. enhancement step or separation step). It will be understood that the enhancement of the first component may be with respect to the second component and / or the suppression of the second component may be with respect to the first component. The first step thereby at least partially separates the first component from any residual components of the audio signal 5. Thus, the first step can also be said to perform a noise removal operation on the audio signal 5.

[0039] As mentioned above, the first step may be to apply speech enhancement to the audio signal 5. In this case, the first component is a speech component and the second component is a noise, background or residual component, etc.

[0040] It is further understood that the first step may be implemented by both traditional audio processing means as well as by deep neural networks, i.e., the first step may in some implementations be a deep learning-based step for emphasizing a first component of the audio signal and / or suppressing a second component of the audio signal relative to the first component.

[0041] Step S1020 is a second step (e.g., a modification or refinement step) of modifying the output of the first step by applying a deep learning-based model to the output of the first step in order to perceptually improve the first component of the audio signal, where the perceptual improvement may relate to (or may include) the removal of distortions and / or artifacts introduced by the first step as well as any remaining undesirable components (e.g., noise or background) not removed by the first step.

[0042] It is understood that step S1010 may be performed by the separator 10 described above, and step S1020 may be performed by the improver 20 described above.

[0043] The first and second steps (similarly, the separator 10 and the improver 20) may operate either in the waveform domain (i.e., directly on the waveform signal) or in the transform domain. One non-limiting example of a transform domain is the spectral domain. In general, the transformation from the waveform domain to the transform domain may involve a time-to-frequency transformation. Thus, the transform domain may also be referred to as the frequency domain.

[0044] When operating in the waveform domain, the first step (similarly, separator 10) receives a time-domain (waveform-domain) audio signal and applies an emphasis of the first component and / or a suppression of the second component to the first component by (directly) modifying the time-domain audio signal. In this case, the output of the first step (similarly, separator 10) is a waveform-domain audio signal in which the first component has been emphasized and / or the second component has been suppressed.

[0045] When operating in the transform domain, the output of the first step (also the separator 10) is a transform domain mask (e.g., a bin-based mask or a band-based mask) indicating weighting coefficients for individual bins or bands of the audio signal. Applying this mask to the (transform domain) audio signal then results in an emphasis of the first component and / or a suppression of the second component relative to the first component. The (transform domain) bins may be, for example, time-frequency bins. Furthermore, the mask may be, for example, a magnitude mask, a phase-sensitive mask, a complex mask, a binary mask, etc. It is understood that the method 1000 may optionally include an (initial) step of transforming the audio signal into the transform domain and / or a (final) step for performing an inverse transform. Similarly, the apparatus described in this disclosure may include a transform stage and an inverse transform stage.

[0046] Returning to Figure 1, the improver 20 receives as input the output 15 of the separator 10. It can take either the single frame output or the multi-frame output from the separator 10.

[0047] For the first option, the improver 20 (as well as the second step) can operate on a single output of the separator 10, as shown in FIG. 1. For example, if the output 15 of the separator 10 is a mask for one frame, the improver 20 will be trained based on this single-frame mask. Here, the separator 10 (if implemented by a deep neural network) and the improver 20 can be trained simultaneously (i.e., in unison), or in a two-stage training strategy described below, where the separator 10 is trained first, and then the training of the improver 20 can follow. In general, it can be said that for the first option, both the training and operation of the second step (as well as the improver) can proceed frame by frame.

[0048] For the second option, the improver 220 (and similarly the second step) can operate on multiple outputs 215 of the separator 210. This situation is illustrated diagrammatically in FIG. 2. For example, the output 215 of the separator 210 can be a mask of one frame, and the improver 220 can be trained based on several outputs 215 of the separator 210. In other words, the outputs 215 of the separator 210 can be ordered / accumulated until a sufficiently large number of frames is available. These multiple outputs 215 of the separator 210 can then be fed to the improver 220 for training. When working with this option, the separator 210 can be trained first (if implemented by a deep neural network).

[0049] In line with the above, for the second option, it can be said that the second step (and thus the improver) receives a plurality of instances of the output of the first step (and thus the separator), each of which corresponds, for example, to a respective one of a plurality of frames of the audio signal. Furthermore, each instance may correspond to a mask for a frame or to a frame of audio. The second step then applies a machine learning-based model to the plurality of instances of the output together to perceptually improve the first component of the audio signal in one or more of the plurality of frames of the audio signal. As mentioned above, the deep learning-based model of the second step may have been trained based on the plurality of instances of the output of the first step and the corresponding plurality of frames of the reference audio signal for the audio signal.

[0050] In another implementation of the second step, the operation and training of the second step may be based on shifting a window of frames that includes the given frame. Thus, the method may maintain a history of previous frames to be considered when generating an output for the given frame. Specifically, in this implementation, the second step receives a sequence of instances of the output of the first step to process the given frame of the audio signal, where each instance corresponds to a respective one in the sequence of frames of the audio signal. It is understood that the sequence of frames includes the given frame. The second step then applies the machine learning-based model together to the sequence of instances of the output to perceptually improve the first component of the audio signal at the given frame. The given frame may be, for example, the most recent frame in the sequence of frames.

[0051] Improver network structure The improver network should depend on the separator design and, in particular, ensure that the separator output matches the improver input. Furthermore, the improver should also be designed based on the specific problems of the separator that need to be addressed (e.g., distortion, artifacts, etc.). A wide range of implementations are available for the improver. The following implementations have been found advantageous for the purpose at hand: 1) an auto-encoder (AE) structure with a bottleneck layer to generate a smooth soft mask in the frequency domain, 2) a recurrent neural network (RNN) / long short-term memory (LSTM) model that allows output of a time-smooth result, and 3) a generative model to recover harmonics lost in the separator component.

[0052] Auto-Encoder based improver Most spectral domain based speech enhancement algorithms suffer from artifacts caused by discontinuity masks, strong / unstable residual noise under low SNR conditions, and residual noise in non-dialog segments. To address these issues, the present disclosure proposes an AE-based improver, as shown diagrammatically in FIG. 3. Thus, a deep learning based model implementing the improver (or, for that matter, the second step) comprises an autoencoder architecture. The autoencoder structure has an encoder stage (or, encoder) 340 and a decoder stage (or, decoder) 360. The encoder 340 and the decoder 360 each include a respective number of successive filter layers 345, 365. The encoder 340 maps the input 315 thereto into a latent space representation 350. The last layer of the encoder 340 may be referred to as a bottleneck layer. The output of the bottleneck layer is the aforementioned latent space representation 350. The decoder 360 maps the latent space representation 350 output by the encoder 340 back to an initial format, i.e., the output 325 of the decoder has the same format as the input 315 to the encoder 340. Thus, the encoder 340 may be said to successively reduce the dimensionality of its input 315 (i.e., from one layer to the next), and the decoder 360 may be said to successively expand the dimensionality of its input (i.e., the latent space representation 350) back to the original dimensionality. Thus, the format of the input / output may correspond to the dimensionality of the input / output. The input 315 to the encoder 340 (i.e., the output of the first step) may be the aforementioned mask, and the output 325 of the decoder 360 may be, for example, a refined mask.

[0053] In one example, the encoder 340 includes multiple successive layers 345 (e.g., DNN layers) having a successively decreasing number of nodes, and the decoder 360 also includes multiple successive layers 365 (e.g., DNN layers) having a successively increasing number of nodes. For example, the encoder 340 and the decoder 360 may have the same number of layers, and the outermost layer of the encoder 340 may have the same number of nodes as the outermost layer of the decoder 360, the next-to-outermost layer of the encoder 340 may have the same number of nodes as the next-to-outermost layer of the decoder 360, and so on down to the respective innermost layers.

[0054] In such an autoencoder structure, the encoder learns an efficient data representation (i.e., latent space representation) of the mask estimated by the separator (as a non-limiting example of the output of the separator) to remove the "mask noise", and the decoder generates an improved mask from the latent representation space by mapping back to the initial space. The improved mask can be smoother and have fewer artifacts due to the mask compression performed by the encoder. Furthermore, such mask reconstruction by the AE-based improver also helps to fix speech distortions and have better discrimination between speech and noise, where better discrimination will help to remove most of the residual noise in non-speech segments.

[0055] A specific non-limiting example of an AE-based implementation of the improver 420 is shown diagrammatically in FIG. 4. The separator 410 in this example is implemented by a multi-scale convolutional neural network operating over the TF domain. For a 48 kHz audio signal 405, the input is transformed into the TF domain by using 4096 short-time Fourier transforms (STFTs) with 50% overlap. The magnitude of 2049 points is then grouped into 1025 bands. Eight frames are fed to the separator 410 (i.e., the input dimensions are 8×1025), and the target is a magnitude mask of one frame (i.e., dimensions 1×1025). The AE-based improver 420 is implemented using a series of DNN layers (e.g., with 512, 256, 512, and 1025 nodes, respectively). The AE-structured encoder learns 256-dimensional representations of the mask, and the decoder reconstructs an improved 1025-dimensional mask by using these 256-dimensional representations. It has been found that such an improver can correct at least some of the target distortion, and can also remove at least some of the residual noise and reduce at least some of the audible artifacts. In general, the perceptual quality is significantly improved by the AE-based improver.

[0056] Recurrent Neural Network Based Improver Considering the temporal discontinuity of some frame-based speech enhancement algorithms, the improver may be implemented using an RNN-based architecture that uses multiple outputs of the separator.

[0057] An example of such an implementation is shown diagrammatically in FIG. 5. The separator may be, for example, the same as the AE-based implementation. The improver 520 comprises several successive layers. In this example, these layers are gated recurrent unit (GRU) / LSTM layers. The separator 520 may be initially trained, for example, with 8 frames as input. At run-time, 32 blocks are fed to the separator, each block containing 8 frames and a frame shift of 1 frame (so that a total of 39 frames are fed to the separator 510). The separator 510 processes the 32 blocks and outputs 32 frame results accordingly. These 32 frame results are then fed as input to the improver 520. The GRU / LSTM-based improver 520 operates on the separator output 515 and helps to improve the speech quality and consistency. The nodes of the GRU / LSTM output layer can be selected to generate the final result. For example, one frame output can be selected based on 32 history frames, or one can choose to generate 32 frame outputs at a time.

[0058] In general, the deep learning based model of the improver (and thus the second step) can implement a recurrent neural network architecture with multiple successive layers, where the multiple layers can be of long short-term memory type or gated recurrent unit type layers.

[0059] Generative model-based improver It has been found that mask-based methods often work well to isolate dominant harmonic components in noisy speech, but may not work well on speech components that are masked / degraded by noise. For example, by using generative models such as waveNet or SampleRNN, it may be possible to reconstruct these missing speech components.

[0060] An example of an implementation of an improver using a generative model is shown diagrammatically in FIG. 6. In this example, a transformer 630 (or ISTFT) is optionally added to convert the output 615 of the separator into the waveform domain, if necessary. The improver 620 then generates a modified audio signal 625 (e.g., a modified dialogue signal) using a series of 1-D dilated convolutionally layers 640 with skip connections 645 followed by a 1-D convolutional layer 660. It can recover the degraded speech components caused by the separator, and can also help remove the residual noise that cannot be removed by the separator.

[0061] In general, the deep learning-based model of the improver (and, similarly, the second step) may implement a generative model architecture having multiple successive convolutional layers, where the convolutional layers may be dilated convolutional layers, optionally including one or more skip connections.

[0062] Training Strategy This disclosure proposes two alternative training strategies for the separator-refiner framework described herein, where it is assumed that the separator and refiner each comprise or implement a deep learning based model, and that training the separator / refiner corresponds to training the respective deep learning based model.

[0063] The first training strategy is a two-stage training strategy. In the first training stage, the separator is trained and its corresponding loss is optimized via back propagation. Once the separator is trained, all its parameters are fixed (i.e., untrainable), and the output of the trained separator is fed to the improver. In the second training stage, only the parameters of the improver are trained, and the loss function of the improver is optimized via back propagation. Thus, the entire framework can be used as the whole model, while the separator and the improver are trained separately in the two training stages. In other words, the improver can be regarded as a deep learning-based customized post-processing block for the separator, which can generally improve the performance of the separator.

[0064] According to the second training strategy, the separator and the improver can be trained simultaneously (i.e., in unison). A challenge and a key issue in doing so may be to ensure that the separator and the improver each perform their own function, i.e., the separator is expected to extract the speech signal, and the improver is expected to improve the performance of the separator. To achieve this goal, a “constrained” training strategy is proposed, in which the loss function used for training considers not only the final output of the improver, but also the intermediate output of the separator. The loss function used for training may be a common loss function for both the deep learning-based model of the separator and the deep learning model of the improver (applied in the first and second steps of the corresponding processing method, respectively). That is, the loss function is based on both the output of the separator and the output of the improver, in addition to appropriate reference data. By considering both the separator loss and the improver loss, the separator can be trained towards dialogue separation (or any desired general audio processing function), and the convergence of the improver is improved as the output of the separator also converges towards the final goal.

[0065] Extending the method Generalizations, extensions, and modifications of the above-described apparatus and methods are now described.

[0066] Multiple separators A number of supervised speech enhancement algorithms have been developed in the past, each with its own advantages and disadvantages. For example, some methods can work well for stationary noise, while others can work well for non-stationary noise. It is difficult to achieve ideal performance for all use cases with only one model of speech enhancement. Therefore, the present disclosure proposes to combine multiple enhancers (i.e., separators) in the framework at hand, as shown diagrammatically in FIG. 7. The system 700 in this implementation comprises multiple separators 710-1, ..., 710-M, which generate outputs output_s1, 715-1 to output_sM, 715-M, respectively. The improver 720 receives these outputs and acts as a "manager" and can fine-tune its performance by comparing and integrating the outputs 715-1, 715-2, ..., 715-M of the separators 710-1, ..., 710-M. Finally, the improver 720 can obtain an aggregate output 725 by reconstructing and weighting the outputs 715-1, 715-2, ..., 715-M of all the separators 710-1, ..., 710-M based on the many-to-one mapping learned during training.

[0067] In general, the above method of audio processing may further include one or more additional first steps for applying an emphasis to a first component of the audio signal and / or applying a suppression to a second component of the audio signal relative to the first component, where the above-mentioned first step and the one or more additional first steps generate mutually (e.g., pairwise) different outputs. For example, the steps may use different models of audio processing (e.g., speech enhancement) and / or different model parameters. The second step then receives the respective outputs of the one or more additional first steps in addition to the output of the first step, and applies its deep learning-based model together to the output of the first step and the output of the one or more additional first steps to perceptually improve the first component of the audio signal. The second step may, among others, apply, for example, weighting and / or selection to the output of the first step and the one or more additional first steps. It will be appreciated that these considerations apply equally to an apparatus (e.g., a system or device) that includes one or more additional separators in addition to a separator and an improver.

[0068] Traditional speech enhancement with deep learning-based improvers A deep learning model structure with a separator and an improver has been proposed above. Conventional (e.g., non-deep learning-based) speech enhancement algorithms cannot be directly embedded into the deep learning model. In order to derive a speech enhancer that is robust to artifacts introduced by conventional methods, the present disclosure proposes a modified framework, as shown in FIG. 8. The framework includes a conventional speech enhancement algorithm together with a deep learning-based improver.

[0069] As can be seen from Fig. 8, a new training strategy can be restored to enable the use of conventional methods as a separator in the proposed framework. Specifically, one or a set of different conventional speech enhancement algorithms can be used, and each of the multiple noise signals in the training set 850 can be used as an input 805 for processing by each of these algorithms (e.g., the separator 810). After that, the enhanced speech signal 815 as well as the original unprocessed noisy speech signal are collected to form a new comprehensive training set 855, which is then used to train a deep learning-based improver 820 to generate an output 825. Here, the unprocessed noise signal and its multiple enhanced versions correspond to the same target speech signal. In other words, the improver 820 tends to learn a many-to-one mapping. As one implementation, the conventional method can be, for example, a Wiener filter based on spectral subtraction, or a priori SNR estimation, etc.

[0070] Improved equipment used in intelligent banding From another perspective, the above-mentioned autoencoder-based improvement can also be considered as related to banding and de-banding processing. In typical signal processing algorithms, more TF characteristics are preserved for higher band numbers, but banding may still be required to reduce processing complexity. However, there are many cases where acceptable performance cannot be achieved by using limited bands when using traditional banding algorithms (e.g., octave bands, 1 / 3 octave bands, etc.). Moreover, it is difficult to evaluate in advance which number of bands should be used to achieve a good tradeoff between complexity and accuracy.

[0071] Regarding the first problem, the aforementioned autoencoder-based refinement can be used to implement the autobanding scheme. The corresponding framework is shown diagrammatically in FIG. 9. Thus, the refiner is split into two parts: a first part (forward) 920-1 for receiving an input 905 (e.g., an input audio signal) and automatically banding it, and a second part (backward) 920-2 for automatically debanding it. Besides the splitting, the same considerations as described above for the autoencoder-based refiner also apply here. That is, the forward refiner 920-1 may include multiple successive layers 930 (e.g., DNN layers) with a successively decreasing number of nodes that eventually map to a latent space representation 935 (code), and the backward refiner 920-2 may also include multiple successive layers 940 (e.g., DNN layers) with a successively increasing number of nodes. For example, the forward refiner and the backward refiner 920-1, 920-2 may have the same number of layers, and the outermost layer of the forward refiner 920-1 may have the same number of nodes as the outermost layer of the backward refiner 920-2, and the next layer of the outermost layer of the forward refiner 920-1 may have the same number of nodes as the next layer of the outermost layer of the backward refiner 920-2, and so on until the innermost layer. The separator 910 may be trained based on the intelligent band features learned by the forward refiner to obtain denoised band features. The denoised band features (i.e., latent space representation 945) are then fed to the back refiner for debanding processing, which finally results in a bin-based output 925.

[0072] Regarding the second problem, the dimensionality of the code (e.g., the latent representation) in the front refiner (i.e., the output by the front refiner) can be modified to determine the most appropriate number of bands. By modifying the dimensionality of the latent representation, performance for different numbers of bands can be evaluated. Thus, the most appropriate number of bands can be selected to provide a good tradeoff between complexity and accuracy.

[0073] As an exemplary implementation, a series of DNN layers (e.g., with 512 nodes and 256 nodes, respectively) can be used by the forward refiner 920-1 to group 1025 points of spectral amplitude (obtained by 2048 points STFT with 50% overlap) and obtain 256-dimensional band features. For the backward refiner 920-2, DNN layers (e.g., with 256 nodes and 512 nodes, respectively) with reverse node number assignment can be used compared with the forward refiner 920-1. The backward refiner 920-2 finally reconstructs a bin-based output (e.g., a bin-based mask) based on the output of the separator (e.g., the denoised band features).

[0074] In general, starting from the method 1000 in FIG. 10 for example, the above-mentioned method of audio processing may further include a third step of applying a deep learning-based model to the audio signal to band the audio signal. The third step should be performed before the first step, so that the order of steps is third step-first step-second step. The second step modifies the output of the first step by debanding the output of the first step. In this configuration, the third step and the second step implement an autoencoder structure for banding and debanding. They can be said to be based on a single deep learning-based model, or alternatively, the deep learning-based models can be said to be jointly trained. As mentioned above, the second step and the third step can respectively implement multiple successive layers with successively increasing and decreasing numbers of nodes.

[0075] In line with the above, one example of a method 1100 of audio processing (e.g., audio enhancement, such as speech enhancement) using intelligent banding is shown generally in the flow chart of Figure 11. Method 1100 may be, for example, a method of speech enhancement. It includes steps S1110 to S1130.

[0076] Step S1110 In this paper, a deep learning based model is applied to an audio signal to band the audio signal.

[0077] Step S1120 In the above, enhancement is applied to a first component of a banded audio signal and / or suppression of the first component is applied to a second component of the banded audio signal relative to the first component.

[0078] Step S1130 In the method, the output of the enhancement step is modified by applying a deep learning based model to the output of the enhancement step in order to deband the output of the enhancement step and to perceptually improve a first component of the audio signal.

[0079] It will be appreciated that the general considerations above for methods of audio processing apply equally to apparatus (eg, systems or devices) for audio processing.

[0080] Generalization to general two-stage neural networks As explained above, the second stage in the proposed framework for audio processing can be an improver to remove artifacts and correct voice distortions. However, the second stage can also have other functions, such as implementing a voice activity detector (VAD). Taking the VAD algorithm as an example, all known VAD algorithms can degrade in accuracy when strong noise is present. It is very challenging for these algorithms to show robust performance in the presence of various noise types and / or for low SNR in general. With the proposed framework, the separator can be used to denoise the mixture (i.e., the input audio signal) and the improver can be used to perform the VAD. Such a VAD system can perform denoising internally and thus become more robust to complex (e.g., noisy) scenarios.

[0081] Thus, the above-mentioned improver may be replaced by an improver that performs deep learning-based VAD on the output of the separator in addition to or as an alternative to removing distortions and / or artifacts, etc.

[0082] Furthermore, the proposed two-step training scheme can be generalized to several other speech enhancement-based applications, such as an equalizer or an intelligibility meter. The separator can perform speech enhancement and background removal as described above, and the improver can be trained based on specific requirements. This can achieve more robust and better results compared to the results when using only the original noisy input of the separator. Thus, the improver can be specifically adapted so that the separator and improver jointly achieve a desired application / operation, such as an equalizer or an intelligibility meter.

[0083] Generalization to multi-stage neural networks in audio processing chains A mature audio signal processing technology chain typically includes several modules (e.g., audio processing modules), some of which may use traditional signal processing methods, and some of which may be based on deep learning. These modules are typically cascaded in series to obtain a desired final output. Based on the proposed framework, each module or part of the modules in such a signal processing chain can be embedded into a large-scale deep learning-based model. When training, each module can be trained in sequence (i.e., separately and in sequence), and the output can be supervised to satisfy the desired result until the end of the last module training. The whole model becomes a deep learning-based audio processing technology chain, and the modules cooperate as expected in the model.

[0084] As such, the present disclosure also relates to any pairing of a signal processing module (e.g., adapted to perform audio processing, audio enhancement, etc.) followed by a deep learning based improver trained to improve the output of the signal processing module. Improving the output of the signal processing module may include one or more of removing artifacts, removing distortion, and / or removing noise.

[0085] Exemplary Computing Device A method of audio processing (e.g., speech enhancement) has been described above. In addition, the present disclosure also relates to an apparatus (e.g., a system or device) for performing the method. One example of such an apparatus is shown in FIG. 1. Further, in line with the method 1000 shown in FIG. 10, the apparatus for processing an audio signal according to the present disclosure can be said to comprise a first stage and a second stage. The first stage and the second stage can be implemented in hardware and / or software. The first stage is adapted to apply an enhancement to a first component of the audio signal and / or a suppression to a second component of the audio signal. The second stage is adapted to modify an output of the first stage by applying a deep learning-based model to the output of the first stage to perceptually improve the first component of the audio signal. Besides, any of the discussions made above can be applied to the first stage and the second stage.

[0086] In general, the present disclosure relates to an apparatus comprising a processor and a memory coupled to the processor, where the processor is adapted to perform steps of the methods described herein. For example, the processor may be adapted to implement the first and second stages described above.

[0087] These aforementioned devices (and stages thereof) may be implemented by a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequentially or otherwise) that specify operations to be performed by the device. Additionally, although only a single device 1400 is shown in the figures, the present disclosure relates to any collection of devices that individually or jointly execute instructions to perform any one or more of the methodologies described herein.

[0088] The present disclosure further relates to a program (eg, a computer program) comprising instructions that, when executed by a processor, cause the processor to perform some or all of the steps of the methods described herein.

[0089] Furthermore, the present disclosure relates to a computer-readable (or machine-readable) storage medium having the aforementioned program stored thereon, where the term "computer-readable storage medium" includes, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media, for example.

[0090] Interpretation and Additional Configuration Considerations The present disclosure relates to methods of audio processing and apparatus (e.g., systems or devices) for audio processing. It is understood that any statements made regarding methods and their steps apply equally and similarly to corresponding apparatus and their stages / blocks / units, and vice versa.

[0091] Unless otherwise indicated, and as will be apparent from the description that follows, throughout this disclosure, descriptions using the terms “processing,” “computing,” “calculating,” “determining,” “analyzing,” and the like, are appreciated to refer to operations and / or processes of a computer or computing system or similar electronic computing device that manipulates and / or transforms data represented as physical quantities, such as electronic quantities, into other data similarly represented as physical quantities.

[0092] Similarly, the term "processor" can refer to any device or portion of a device that processes electronic data, e.g., from registers and / or memory, and converts the electronic data into other electronic data that can be stored, e.g., in registers and / or memory. A "computer" or a "computing machine" or a "computing platform" can include one or more processors.

[0093] The methods described herein are executable by one or more processors that, in one exemplary embodiment, accept computer-readable (also referred to as machine-readable) code that includes a set of instructions that, when executed by one or more of the processors, perform at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken is included. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem including a main RAM, and / or a static RAM, and / or a ROM. A bus subsystem may be included for communicating between the components. The processing system may further be a distributed processing system with the processors coupled by a network. In the case where the processing system requires a display, such a display may include, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. In the case where manual data entry is required, the processing system also includes input devices such as one or more of an alphanumeric input unit, such as a keyboard, a pointing control device, such as a mouse, and the like. The processing system may also include a storage system, such as a disk drive unit. The processing system in some configurations may include a sound output device, and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium that carries computer-readable code (e.g., software) that includes a set of instructions that, when executed by one or more processors, cause the execution of one or more of the methods described herein. Note that when a method includes several elements, e.g., several steps, no ordering of such elements is implied unless specifically stated.The software may reside in the hard disk, and may also reside completely or at least partially in the RAM and / or in the processor during execution by the computer system. Thus, the memory and the processor also constitute a computer readable carrier medium carrying computer readable code. Moreover, the computer readable carrier medium may form or be included in a computer program product.

[0094] In alternative exemplary embodiments, one or more processors may operate as stand-alone devices or may be connected, e.g., networked, to other processors in a networked deployment, where one or more processors may operate in the capacity of a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) that specify operations to be performed by that machine.

[0095] Note that the term "machine" should also be interpreted to include any collection of machines that individually or collectively execute a set (or sets) of instructions for performing any one or more of the methodologies described herein.

[0096] Thus, one exemplary embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program for execution on one or more processors, e.g., one or more processors that are part of a web server configuration. Thus, as will be appreciated by those skilled in the art, the exemplary embodiments of the present disclosure may be embodied as a method, an apparatus, e.g., a dedicated device, an apparatus, e.g., a data processing system, or a computer-readable carrier medium, e.g., a computer program product. The computer-readable carrier medium carries computer-readable code that includes a set of instructions that, when executed on one or more processors, causes the processor to perform the method. Thus, aspects of the present disclosure may take the form of a method, an entirely hardware exemplary embodiment, an entirely software exemplary embodiment, or an exemplary embodiment that combines software and hardware aspects. Furthermore, the present disclosure may take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

[0097] The software may also be transmitted or received over a network via a network interface device. Although the carrier medium is a single medium in an exemplary embodiment, the term "carrier medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "carrier medium" should also be interpreted to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by one or more of the processors and causing one or more of the processors to perform any one or more of the methods of the present disclosure. The carrier medium can take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a bus subsystem. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" thus includes, but is not limited to, computer products embodied in solid-state memories, optical and magnetic media, media carrying a propagated signal detectable by at least one processor or one or more processors and representing a set of instructions that, when executed, implement a method, and transmission media in a network carrying a propagated signal detectable by at least one processor of one or more processors and representing a set of instructions.

[0098] It will be understood that the steps of the described method are, in one exemplary embodiment, performed by a suitable processor of a processing (e.g., computer) system executing instructions (computer-readable code) stored in storage. It will also be understood that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functions described herein. The present disclosure is not limited to any particular programming language or operating system.

[0099] Throughout this disclosure, references to "one example embodiment," "some example embodiments," or "an example embodiment" mean that a particular feature, structure, or characteristic described in connection with an example embodiment is included in at least one example embodiment of the disclosure. Thus, the appearances of the phrases "in one example embodiment," "some example embodiments," or "in an example embodiment" in various places throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments.

[0100] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe a common object merely indicates that different instances of a similar object are being referred to and is not intended to imply that the objects so described must be in a given order in time, space, ranking, or in any other manner.

[0101] In the following claims and in the description of this specification, any one of the terms comprising, comprised of, and which comprises is an open term meaning that the term includes at least the following element / feature, but does not exclude others. Thus, the term comprising, when used in the claims, should not be interpreted as being limited to the means or elements or steps listed thereafter. For example, the scope of a device comprising A and B should not be limited to a device consisting of elements A and B only. As used herein, any one of the terms including, which includes, or that includes is also an open term meaning that the term includes at least the following element / feature, but does not exclude others. Thus, including is synonymous with and means comprising.

[0102] In the above description of exemplary embodiments of the present disclosure, it should be understood that various features of the present disclosure may be grouped together in a single exemplary embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in understanding one or more of the various inventive aspects. This method of disclosure, however, should not be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed exemplary embodiment. Accordingly, the claims following the specification are expressly incorporated into this specification, with each claim standing on its own as a separate exemplary embodiment of the present disclosure.

[0103] Furthermore, although some exemplary embodiments described herein include features that are not included in other exemplary embodiments, as will be understood by one of ordinary skill in the art, combinations of features of different exemplary embodiments are intended to be within the scope of the present disclosure and form different exemplary embodiments. For example, in the following claims, any of the claimed exemplary embodiments may be used in any combination.

[0104] In the description provided herein, numerous specific details are set forth. However, it is understood that exemplary embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0105] Thus, while what is believed to be the best mode of the disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the disclosure, and it is intended to claim all such changes and modifications that fall within the scope of the disclosure. For example, any formulas given above are merely representative of procedures that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted to methods described within the scope of the disclosure.

[0106] Various aspects of the present invention can be understood from the exemplary embodiments (EEE) listed below.

[0107] EEE1. A method for processing an audio signal, comprising: This is the first step, applying emphasis to a first component of the audio signal and / or applying suppression to a second component of the audio signal relative to the first component; The first step, a second step of modifying the output of the first step by applying a deep learning based model to the output of the first step; removing artifacts and / or distortions introduced into the audio signal by the first step and perceptually enhancing the first component of the audio signal. A second step.

[0108] EEE2. The method of EEE1, wherein the first step is applying speech enhancement to the audio signal.

[0109] EEE3. The output of the first step is the audio signal in the waveform domain, The method according to EEE1 or 2, wherein the first component is enhanced and / or the second component is suppressed relative to the first component.

[0110] EEE4. The output of the first step is a mask in the transform domain indicating weighting coefficients for each bin or band; and The method according to any one of claims 1 to 2, wherein applying the mask to the audio signal results in an emphasis of the first component and / or a suppression of the second component relative to the first component.

[0111] A second step receives a plurality of instances of the output of the first step; each of the instances corresponds to a respective one of a plurality of frames of the audio signal; and The method of any one of EEE1 to 4, wherein the second step comprises jointly applying a machine learning based model to a plurality of instances of the output to perceptually refine the first component of the audio signal in one or more of a plurality of frames of the audio signal.

[0112] EEE6. The second step comprises receiving, for a given frame of the audio signal, a sequence of instances of the output of the first step; each of the instances corresponds to a respective one in a sequence of frames of the audio signal; the sequence of frames includes the given frame, and The method of any one of EEE1 to EEE5, wherein the second step comprises jointly applying the machine learning based model to a sequence of instances of the output to perceptually improve the first component of the audio signal at the given frame.

[0113] EEE7. The deep learning based model of the second step implements an auto-encoder architecture having an encoder stage and a decoder stage; Each stage comprises a respective plurality of successive filter layers; The encoder stage maps an input to the encoder stage into a latent space representation; and The method of any one of EEE1-6, wherein the decoder stage maps the latent space representation output by the encoder stage to an output of the decoder stage having the same format as the input to the encoder stage.

[0114] EEE8. The deep learning based model of the second step implements a recurrent neural network architecture having multiple successive layers; The method of any one of EEE1 to EEE6, wherein the layers are long short-term memory type or gated recurrent unit type layers.

[0115] EEE9. The method of any one of EEE1 to EEE6, wherein the deep learning based model implements a generative model architecture having multiple successive convolutional layers.

[0116] EEE10. The method of EEE9, wherein the convolutional layer is a dilated convolutional layer, optionally including skip connections.

[0117] EEE11. The method further comprising: one or more additional first steps for applying emphasis to the first component of the audio signal and / or for applying suppression to the second component of the audio signal, the first step and the one or more additional first steps generate mutually distinct outputs; the second step receives, in addition to the output of the first step, an output of each of the one or more additional first steps; and The method of any one of EEE1 to 10, wherein the second step applies the deep learning based model jointly to the output of the first step and to the output of the one or more additional first steps to perceptually refine the first component of the audio signal.

[0118] EEE12. The method further comprising: a third step of applying a deep learning based model to the audio signal to band the audio signal prior to input to the first step; the second step modifies the output of the first step by debanding the output of the first step; and The method of any one of EEE1 to 11, wherein the deep learning based models of the second step and the third step are trained together.

[0119] EEE13. The method of EEE12, wherein the second and third steps implement a plurality of successive layers having successively increasing and decreasing numbers of nodes, respectively.

[0120] EEE14. The first step includes applying a deep learning based model to enhance the first component of the audio signal and / or suppress the second component of the audio signal relative to the first component. A method according to any one of EEE1 to 13.

[0121] EEE15. Apparatus for processing audio signals, comprising: This is the first stage, applying emphasis to a first component of the audio signal and / or applying suppression to a second component of the audio signal relative to the first component; The first stage and a second stage that modifies the output of the first stage by applying a deep learning based model to the output of the first stage; removing artifacts and / or distortions introduced into the audio signal by the first stage and perceptually enhancing the first component of the audio signal. a second stage.

[0122] EEE16. An apparatus including a processor and a memory coupled to said processor, The apparatus, wherein the processor is adapted to perform the steps of the methods described in any one of EEE1 to EEE14.

[0123] EEE17. A computer program comprising instructions, A computer program which, when executed by a computing device, causes the computing device to perform the steps of the methods described in any one of EEE1 to EEE14.

[0124] EEE18. A computer readable storage medium storing a computer program according to EEE17.

Claims

1. 1. A method for processing an audio signal, comprising the steps of: - a first step of applying emphasis to a first component of the audio signal and / or applying suppression to a second component of the audio signal relative to the first component; a second step of modifying the output of the first step by applying a deep learning based model to the output of the first step to perceptually improve the first component of the audio signal; A method comprising:

2. The first step is to apply speech enhancement to the audio signal. The method of claim 1.

3. the output of said first step is a waveform-domain audio signal in which said first component is emphasized and / or said second component is suppressed relative to said first component; The method according to claim 1 or 2.

4. the output of the first step is a transform domain mask indicating weighting factors for individual bins or bands, and applying the mask to the audio signal results in an emphasis of the first component and / or a suppression of the second component relative to the first component. The method according to claim 1 or 2.

5. A second step receives a plurality of instances of the output of the first step; each of the instances corresponds to a respective one of a plurality of frames of the audio signal; and The second step includes jointly applying a machine learning based model to the output instances to perceptually refine the first component of the audio signal in one or more of the frames of the audio signal.

5. The method according to any one of claims 1 to 4.

6. the second step receiving, for a given frame of the audio signal, a sequence of instances of the output of the first step; each of the instances corresponds to a respective one in a sequence of frames of the audio signal; the sequence of frames includes the given frame, and the second step jointly applying the machine learning based model to a sequence of instances of the output to perceptually enhance the first component of the audio signal at the given frame; The method according to claim 5.

7. The deep learning based model of the second step implements an auto-encoder architecture having an encoder stage and a decoder stage; Each stage comprises a respective plurality of successive filter layers; The encoder stage maps an input to the encoder stage into a latent space representation; and the decoder stage maps the latent space representation output by the encoder stage to an output of the decoder stage having the same format as the input to the encoder stage.

7. The method according to any one of claims 1 to 6.

8. the deep learning based model of the second step implements a recurrent neural network architecture having multiple successive layers; The layers are of long short-term memory type or gated recurrent unit type.

7. The method according to any one of claims 1 to 6.

9. The deep learning based model implements a generative model architecture having multiple successive convolutional layers.

7. The method according to any one of claims 1 to 6.

10. the convolutional layer is a dilated convolutional layer, optionally including skip connections; 10. The method of claim 9.

11. The method further comprises: one or more additional first steps for applying emphasis to the first component of the audio signal and / or for applying suppression to the second component of the audio signal, the first step and the one or more additional first steps generate mutually distinct outputs; the second step receiving, in addition to the output of the first step, an output of each of the one or more additional first steps; and the second step jointly applying the deep learning based model to the output of the first step and to the output of the one or more additional first steps to perceptually refine the first component of the audio signal.

11. The method according to any one of claims 1 to 10.

12. The method further comprises: a third step of applying a deep learning based model to the audio signal to band the audio signal prior to input to the first step; the second step modifies the output of the first step by debanding the output of the first step; and The deep learning based models of the second and third steps are trained together.

12. The method according to any one of claims 1 to 11.

13. the second step and the third step respectively implement a plurality of successive layers having successively increasing and decreasing numbers of nodes, The method of claim 12.

14. the first step being a deep learning based step for emphasizing the first component of the audio signal and / or suppressing the second component of the audio signal relative to the first component.

14. The method according to any one of claims 1 to 13.

15. 1. An apparatus for processing an audio signal, comprising: a first stage for applying emphasis to a first component of the audio signal and / or for applying suppression to a second component of the audio signal relative to the first component; a second stage for modifying the output of the first stage by applying a deep learning based model to the output of the first stage to perceptually improve the first component of the audio signal; 13. An apparatus comprising:

16. 1. An apparatus comprising: a processor; and a memory coupled to the processor, The processor is adapted to carry out the steps of the method according to any one of claims 1 to 14. Device.

17. A computer program comprising instructions, When executed by a computing device, the computing device performs the steps of the method according to any one of claims 1 to 14. Computer program.

18. 18. A computer program according to claim 17, A computer-readable storage medium.

Images