PCEN-based mask thresholding and sound activity detection for training DNN-based speech enhancement models

Abstract

The present disclosure describes a method of determining at least one mask for training a deep neural network (DNN)-based mask-based audio processing model. In particular, the method may include obtaining a time-frequency representation of a target audio signal for training. The method may also include determining a channel-by-channel energy normalization (PCEN) metric for the target audio signal. The method may also further include determining at least one mask based on the PCEN metric.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 493,979 filed on April 3, 2023, U.S. Provisional Application No. 63 / 437,273 filed on January 5, 2023, and International Application No. PCT / CN2022 / 087983 filed on April 20, 2022, each of which is incorporated herein in its entirety. Technical Field

[0003] The present disclosure is directed to the general field of audio processing and, more specifically, to speech enhancement using neural networks. Background Art

[0004] Generally speaking, the processing related to speech enhancement usually aims to remove one or more unwanted artifacts from a noisy waveform and retain the clean speech. Recently, such speech enhancement processes also tend to utilize machine learning-based techniques, such as deep neural networks (DNNs).

[0005] Broadly speaking, in an exemplary DNN-based system, a noisy waveform, which may be a mixture of clean speech, noise, echo, reverberation, and a far-end echo reference signal, may be processed by spectral transformation and (optionally) banding. The DNN may then take the mixed band energies as input and attempt to predict a mask for each time-frequency band. Mask values ​​are typically between 0 and 1, which typically represent the (estimated / predicted) degree of speech present in the mixture. During training, the DNN may be exposed to a large mixed dataset, which may be manually created, for example, by randomly adding all signal components or similar methods. In some possible implementations, for each sample, an ideal mask—such as an ideal ratio mask (IRM)—may be defined according to the following formula:

[0006]

[0007] Among them, S(t, f), N(t, f) and E(t, f) are generally the time-frequency band energies of clean speech, noise and echo, respectively. Then, as the technician can understand and recognize, the weights of the DNN will be updated in the direction of minimizing the loss function, which usually measures the difference between the predicted mask and the ideal mask.

[0008] However, there may be some problems with the above-mentioned ideal mask-based techniques. First, it may often be challenging to collect a large amount of studio-quality clean speech signals. Many times, (before adding additional noise during training, e.g., for denoising tasks) the "clean" signal may carry a certain degree of stationary noise, such as recording noise, room noise, etc. As a result, the IRM is usually larger than it should be, and models trained with such IRMs are usually unable to remove the associated stationary noise. In addition, in the clean speech signal, there may be some time-frequency bands whose energy is relatively small (e.g., below a certain threshold). Omitting these bands may not be perceptually noticeable. However, an algorithm is needed to carefully select the threshold to avoid zeroing out meaningful speech. In addition, in some possible cases, the speech enhancement DNN may also need to (at least implicitly) learn the behavior of a voice activity detector (VAD), for example in order to be able to retain speech and remove noise in speech areas, while (actively) removing unwanted artifacts in non-speech areas.

[0009] In view of some or all of the above issues, in general, there is a need for techniques and / or mechanisms to perform voice activity detection and / or mask thresholding for training DNN-based speech enhancement models. Summary of the invention

[0010] In view of the above, the present disclosure as a whole provides: a method for determining at least one mask, wherein the at least one mask is used to train a mask-based audio processing model based on a deep neural network (DNN); a method for determining a loss function for speech perception, wherein the loss function is used to train a DNN-based audio processing model; a method for classifying a target audio signal into a speech frame or a non-speech frame; corresponding devices, programs, and computer-readable storage media, which have the features of the corresponding independent claims.

[0011] According to a first aspect of the present disclosure, a method for determining at least one mask (e.g., one or more masks) is provided, which is used to train a mask-based audio processing model based on a DNN. The model can be trained to perform any suitable audio-related processing, such as (but certainly not limited to) speech enhancement (e.g., denoising, removing artifacts, etc.). As will also be understood and appreciated by the skilled person, any suitable DNN (or more generally, any suitable neural network) implementation may be used regardless of the actual architecture, depending on various implementations and / or applications.

[0012] In particular, the method may include obtaining a time-frequency representation of a target audio signal for training. In general, particularly in the technical context of the present disclosure, unless otherwise specified, the term "target audio signal" is generally used to refer to an audio signal for which a DNN-based model is trained to perform corresponding audio processing. As an illustrative (non-limiting) example that is easy to understand, in some cases where the model is trained to retain only clean audio signals from a noisy audio mixture (e.g., by removing all unwanted artifacts), the target audio signal may be a clean audio (e.g., speech) signal (e.g., collected by a professional). In some other cases where the model is trained not to remove reverberation (but to remove all other artifacts) from a noisy audio mixture, the target audio signal may be a reverberant audio signal (i.e., with reverberation). Therefore, in some possible cases, such a target audio signal may also be understood as representing a groundtruth for training.

[0013] The method may also include determining a per-channel energy normalization (PCEN) metric for the target audio signal. As will be discussed in more detail below, PCEN may be viewed as an audio signal processing technique for normalizing the energy of each frequency channel in a spectrogram. Broadly speaking, PCEN-based techniques may seek to normalize the energy of each frequency channel in a spectrogram by dividing the energy of the channel by an amount proportional to the average energy of the channel over time. This helps reduce the effects of background noise and signal level variations across different frequency channels.

[0014] Accordingly, the method may further include determining at least one mask based on the PCEN metric.

[0015] Configured in the above manner, the proposed method can generally provide an efficient, flexible but reliable mechanism for determining masks that may be used later to train DNN-based audio processing (e.g., speech enhancement) models. Specifically, as mentioned above, the "clean" speech signals collected for training (e.g., S(t,f) in the above formula (1)) often have a certain degree of stationary noise, which may cause the DNN trained using conventional IRM to be unable to successfully and completely remove such stationary noise. Now, in particular, by introducing the PCEN metric as proposed above into the process, (one or more) masks can be determined in a way that can also largely (in some cases even completely) remove such stationary noise (usually embedded in the "clean" signal), thereby significantly improving the performance of the subsequent training process, and thus, significantly improving the final performance in real practice (i.e., after the model has been trained). Incidentally, it should also be noted that, compared with conventional techniques, the PCEN metric proposed in the present disclosure is usually only used in the training phase and not in testing.

[0016] In some example embodiments, the target audio signal may include speech and / or music.As described above, as will be appreciated and understood by a skilled person, the target audio signal may include any other suitable form of an audio signal depending on different embodiments and / or applications.

[0017] In some example embodiments, the target audio signal may be used as a ground truth for training.

[0018] In some example embodiments, the target audio signal may include a clean audio component and a stationary noise component, such as recording noise and / or room noise. As described above, in some possible cases, such as during studio production, the obtained "clean" signal may (sometimes inevitably) include a certain degree of stationary noise, which may include (but is certainly not limited to) recording noise (e.g., mechanical noise of the recording equipment), (background) room noise (e.g., airflow sound, etc.). However, it is worth mentioning that, as can be understood and recognized by the technician, especially in the technical context of the present disclosure, such stationary noise should be distinguished from those (e.g., artificial / artificially generated) noises that may have been specifically or even deliberately inserted or added to the "clean" signal for the purpose of training (e.g., denoising). In addition, it may be worth emphasizing again that (in the case where the "clean" audio signal is used as a reference truth value) the training process itself is usually to eliminate those artificial / artificially generated noises; on the other hand, the technology proposed by the present invention is usually to remove the effects or influences of these stationary noises (which may already exist in the "clean" audio signal used as the reference truth value for training), thereby improving the performance of the overall training.

[0019] In some example embodiments, the PCEN metric can be determined based on the ratio between the time-frequency energy metric of the target audio signal and the running average of the time-frequency energy metric of the target audio signal. As the technician can understand and recognize, similar to the situation of time-frequency representation, any suitable time-frequency energy metric can also be used here according to various embodiments and / or situations. For example, in some possible cases, the time-frequency energy metric can be implemented as band energy. In some other possible cases, for example, when not using banding, the time-frequency energy metric can be realized using, for example, binning (bin) energy instead. Of course, this generality is applied to any suitable transformation (for example, fast Fourier transform (FFT), modulation discrete transform (MDXT), complex quadrature mirror filter (CQMF) etc.) and / or banding type.

[0020] In some example embodiments, the method may further include obtaining an audio mixture (or sometimes also referred to as an audio mix-up) based on the target audio signal. In particular, in addition to the target audio signal, the audio mixture may include audio artifacts - for example, noise, (far-end) echo, reverberation, etc. Depending on different situations, the audio mixture may be pre-collected (for example, from one or more available databases) or generated (for example, manually or automatically). As can be understood and appreciated by the technician, the audio mixture may be generated for the purpose of complying with the corresponding training (corresponding to the corresponding audio processing aspects). As an illustrative (non-limiting) example, when training for noise removal (denoising), the audio mixture may be generated in the form of including various suitable (for example, artificially generated) noises. Similarly, as another illustrative (non-limiting) example, when training for reverberation elimination, the audio mixture may be generated in the form of including various suitable (for example, artificially generated) reverberations. More specifically, determining at least one mask based on the PCEN metric may include determining at least one ideal ratio mask (IRM) metric based on the target audio signal and the audio mixture; and adjusting at least one IRM metric to obtain at least one mask based on the PCEN metric. In other words, it can be understood that the technology suggested by the present disclosure can use the IRM metric as a starting point, and by appropriately and properly adjusting the IRM metric based on the PCEN metric (this feature (together with other factors) reduces or eliminates the impact of possible stationary noise hidden in the clean reference truth audio signal), a final mask that can be used for subsequent training processes is obtained. As can be understood and appreciated by a technician, the IRM metric can be determined according to the above formula (1) or by using any other suitable implementation.

[0021] In some example embodiments, the audio artifact may include at least one of: noise, echo, or reverberation. As described above, such artifact-like noise is to be distinguished from the above-mentioned stationary noise that may be hidden in the clean reference truth audio signal. Of course, as will be understood and appreciated by the skilled person, any other suitable audio artifact may also be considered for training purposes.

[0022] In some example embodiments, the IRM metric may be determined as a ratio of a time-frequency energy metric of the target audio signal to a time-frequency energy metric of an audio mixture including audio artifacts, such as according to formula (1) defined above. As indicated above, any suitable implementation of the time-frequency energy metric may be considered for such calculations. Of course, any other suitable manner may be used to determine (calculate) according to various embodiments and / or applications.

[0023] In some example embodiments, the corresponding values ​​of the PCEN and IRM metrics may be determined for each time-frequency band of the target audio signal. Thus, the adjustment of the IRM metric may include: for each time-frequency band, if the corresponding PCEN metric value is less than a first predetermined (or preconfigured) threshold value (e.g., 10 -5 ), the corresponding IRM metric value of the time-frequency band is set to 0. In particular, by setting the IRM metric of the frequency band with stationary noise to 0, the model can be trained to learn to completely remove such (stationary) noise (which may be difficult to identify and / or remove by other means). In addition, by omitting the small energy speech band by setting its IRM metric to 0, the variability of the training target can be reduced, and the model may also be able to focus on meaningful speech bands. More specifically, considering that the PCEN metric generally represents a smoothed version of the target signal, the energy dynamic range after PCEN will be relatively narrow (compared to techniques that do not use PCEN). Therefore, a global constant can be appropriately determined as a threshold. For example, in some possible (non-limiting) embodiments, the threshold can be obtained by generating many histograms of the energy of the speech bands processed by PCEN and selecting (meaningful) small boundary values ​​that will not output zero speech.

[0024] In some example embodiments, the method may further include classifying the target audio signal as a speech frame or a non-speech frame. Therefore, the method may further include, for each audio frame, determining at least one mask further based on the classification of the speech frame or the non-speech frame.

[0025] In some example embodiments, for each audio frame of the target audio signal, classifying the target audio signal as a speech frame or a non-speech frame may include: determining a corresponding frame-by-frame PCEN metric across all frequency bands of the audio frame; if the determined frame-by-frame PCEN metric is greater than a second predetermined threshold, classifying the audio frame as a speech frame; otherwise, classifying the audio frame as a non-speech frame. As can be understood and appreciated by a skilled person, the (previously mentioned) PCEN metric and the frame-by-frame PCEN metric and / or the first and second predetermined thresholds may (but certainly not necessarily) be the same or calculated in different ways. In some possible embodiments, the second predetermined threshold may be empirically determined as the first predetermined threshold multiplied by the number of frequency bands in the frame (e.g., an average value).

[0026] In some example embodiments, determining at least one mask further based on the classification of speech frames or non-speech frames may include setting corresponding IRM metric values ​​of all non-speech frames to 0. In other words, if the frame energy is below a second threshold, the frame may be classified as a non-speech frame, and thus, corresponding IRM values ​​of all frequency bands in the frame may be set to 0, which may help to further reduce the variability of the training objectives.

[0027] In some example embodiments, for each time-frequency band, the PCEN metric may be determined according to the following formula:

[0028]

[0029] Where S(t,f) is the time-frequency energy of the target audio signal, ε, α, δ and r are predetermined constants, and M(t,f) is the running average of S(t,f), which is defined as:

[0030] M(t,f)=(1-s)·M(t-1,f)+s·S(t,f),

[0031] Wherein, s is a predetermined smoothing factor. It should be noted that this calculation is given only as an illustrative example and should not be construed as any form of limitation. Of course, as can be understood and appreciated by a skilled person, any other suitable manner may be used to determine or calculate the PCEN metric according to various embodiments and / or applications.

[0032] In some example embodiments, the mask-based audio processing model based on the DNN may be a multi-task model configured to be trained for predicting a plurality of masks, each of which corresponds to a corresponding audio processing aspect. That is, in some possible embodiments, the DNN-based model may be trained to predict K sets of labels. As an illustrative example, one mask may be defined to remove any artifacts (such as noise, reverberation, echo, etc.), while a second mask may be trained not to remove reverberation (but to remove all other artifacts), and finally, a third mask may be used to retain music and speech. Thus, the method may include the following steps: obtaining a corresponding target audio signal for each audio processing aspect; determining a corresponding PCEN metric for each target audio signal; and determining a corresponding mask based on the PCEN metric. However, it should be noted that, in a broad sense, one or more PCEN-based thresholds shown above may be applied to one or more sets of masks. In other words, some masks may (but not necessarily have to) use the proposed PCEN-based technique, while others do not.

[0033] According to a second aspect of the invention, there is provided a method for determining a loss function for speech perception, which is used to train a mask-based audio processing model based on a DNN. The model can be trained to perform any suitable audio-related processing, such as (but certainly not limited to) speech enhancement (e.g., denoising, removing artifacts, etc.), etc. As will be understood and appreciated by the skilled person, any suitable DNN (or more generally, neural network) implementation may be used regardless of the actual architecture, depending on the various implementations and / or applications.

[0034] In particular, the method may include obtaining a time-frequency representation of a target audio signal for training. In general, particularly in the technical context of the present disclosure, unless otherwise specified, the term "target audio signal" is generally used to refer to an audio signal for which a DNN-based model is trained in order to perform corresponding audio processing. As an illustrative (non-limiting) example that is easy to understand, in some cases where the model is trained to retain only clean audio signals from a noisy audio mixture (e.g., by removing all unwanted artifacts), the target audio signal may be a clean audio (e.g., speech) signal (e.g., collected by a professional). In some other cases where the model is trained to not remove reverberation (but remove all other artifacts) from a noisy audio mixture, the target audio signal may be a reverberant audio signal (i.e., with reverberation).

[0035] The method may also include determining the presence of speech in the audio signal by using a voice activity detection (VAD) process. As will be appreciated and appreciated by the skilled person, any suitable (existing or new) VAD technique may be used. An illustrative (non-limiting) example of an existing VAD technique may be a conventional WebRTC VAD technique.

[0036] The method may also include determining a loss function based on the presence of speech in the determined target audio signal. In particular, as will be appreciated and appreciated by a skilled person, the loss is typically minimized using gradient descent by back-propagating gradients through all model weights. Therefore, determining a loss function based on the presence of speech in the determined target audio signal may be achieved by controlling the gradient of the loss function.

[0037] In some example embodiments, the VAD process may include determining the presence of speech in the target audio signal frame by frame and / or frequency band by frequency band. In addition, controlling the gradient of the loss function based on the determined presence of speech in the target audio signal may include increasing the corresponding gradient of the loss function for non-speech frames and / or non-speech bands, thereby more actively suppressing audio artifacts in the non-speech frames and / or non-speech bands. That is, in some possible embodiments, the gradient may be controlled so that the predicted mask is actively driven toward 0, thereby removing unwanted artifacts in non-speech frames.

[0038] Configured in the above manner, the proposed method can generally provide an efficient, flexible but reliable mechanism to determine a loss function, which can be used to train a DNN-based audio processing (e.g., speech enhancement) model later. Specifically, according to the above-proposed technique, the loss function can be determined in a design that assigns a higher priority to retaining speech (in speech frames) than to removing noise; while in non-speech frames, noise (and other unwanted artifacts, such as echoes, etc.) can be suppressed as aggressively as possible. Therefore, the performance of the training process can be significantly improved, and thus the final performance in real practice (i.e., after the model is trained) can be significantly improved.

[0039] In some example embodiments, for each audio frame of the target audio signal, determining the frame-by-frame presence of speech in the target audio signal comprises: determining a frame-by-frame per-channel energy normalization (PCEN) metric across all frequency bands of the corresponding audio frame; and if the determined frame-by-frame PCEN metric is greater than a predetermined threshold, determining that the corresponding audio frame contains speech. The frame-by-frame PCEN may be determined (e.g., calculated) in any suitable manner, e.g., similar to or different from the manner described above with respect to the first aspect of the present disclosure.

[0040] In some example embodiments, for each speech frame of the target audio signal, determining the presence of speech in the audio signal may include: for each frequency band of the speech frame, determining a corresponding frequency band-by-frequency PCEN metric; and if the determined frequency band-by-frequency PCEN metric is less than a second predetermined threshold, determining the corresponding frequency band of the speech frame as a non-speech band. Similar to the above, the frequency band-by-frequency PCEN metric may be determined or calculated in the same or different manner as described above with respect to the first aspect of the present disclosure.

[0041] In some example embodiments, the loss function may be determined such that the penalty for over-suppression of speech in the target audio signal is greater than the penalty for under-suppression of noise. Generally speaking, one of the main reasons for this is that (speech enhancement related) DNNs are usually trained to preserve speech and remove noise in speech areas (often tending to preserve speech), while actively removing unwanted artifacts in non-speech areas (without caring about over-suppression of speech).

[0042] In some example embodiments, for each time-frequency band, the loss function loss may be defined as:

[0043] loss = a diff -diff-1,

[0044] Where a is a predetermined constant, and diff indicates the difference between the ideal ratio mask (IRM) metric determined for the target audio signal and the estimated mask mask predicted by the DNN-based audio processing model est Of course, as will be understood and appreciated by the skilled person, any suitable loss function may be determined / designed depending on various implementations and / or applications.

[0045] In some example implementations, for a speech frame, diff may be defined as:

[0046]

[0047] And for non-speech frames, diff can be defined as:

[0048]

[0049] Wherein γ is a predetermined constant. As described above, according to various implementations and / or applications, any other suitable manner may be used for determination or calculation.

[0050] In some example embodiments, the IRM metric may be a mask determined by using the method according to any one of the example embodiments set forth with respect to the first aspect of the present disclosure.

[0051] In some example embodiments, determining the loss function may include: determining corresponding loss functions for speech frames and non-speech frames; and averaging the loss functions of all time-frequency bands to obtain a final loss function. Of course, as described above, any other suitable method may be used to determine or calculate the loss function according to various embodiments and / or applications.

[0052] According to a third aspect of the present invention, a method for training an audio processing model based on a deep neural network (DNN) is provided. In particular, the DNN-based audio processing model can be trained based on a mask and / or a (speech-aware) loss function, wherein the mask is determined by using the method according to any one of the example embodiments proposed in relation to the first aspect of the present disclosure, and the (speech-aware) loss function is determined by using the method according to any one of the example embodiments proposed in relation to the second aspect of the present disclosure.

[0053] According to a fourth aspect of the present invention, a method for classifying a target audio signal as a speech frame or a non-speech frame is provided. In particular, for each audio frame of the target audio signal, the method may include: determining a corresponding frame-by-frame channel energy normalization (PCEN) metric across all frequency bands of the audio frame; and if the determined frame-by-frame PCEN metric is greater than a predetermined threshold, classifying the audio frame as a speech frame; otherwise classifying the audio frame as a non-speech frame.

[0054] According to a fifth aspect of the present invention, there is provided an apparatus comprising a processor and a memory coupled to the processor. The processor may be adapted to cause the apparatus to perform all steps according to any example method described in the preceding aspects and any example implementations thereof.

[0055] According to a sixth aspect of the present invention, a computer program is provided. The computer program may include instructions, which, when executed by a processor, cause the processor to perform all the steps of the example method described in the entire disclosure.

[0056] According to a seventh aspect of the present invention, a computer-readable storage medium is provided, which can store the aforementioned computer program.

[0057] It should be appreciated that device features and method steps can be interchanged in a variety of ways. In particular, as will be appreciated by a skilled person, details of the disclosed method(s) may be implemented by a corresponding device (or system), and vice versa. Furthermore, any of the above statements regarding the method(s) are to be understood to apply equally to the corresponding device (or system), and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Example embodiments of the present disclosure will be explained below with reference to the accompanying drawings, wherein like reference numerals indicate like or similar elements, and wherein:

[0059] Figure 1 is a schematic diagram showing an example audio processing system based on a deep neural network (DNN),

[0060] Figure 2 is a schematic diagram showing a comparison between example frequency band energies of a speech signal with stationary noise and its version based on per-channel energy normalization (PCEN),

[0061] Figure 3 is a schematic diagram showing an example implementation of a loss function according to an embodiment of the present disclosure,

[0062] Figure 4 is a schematic diagram showing another example implementation of a loss function according to an embodiment of the present disclosure,

[0063] Figure 5 is a schematic diagram showing a diagram of an example DNN configured for training a multi-task model according to an embodiment of the present disclosure,

[0064] Figure 6 is a schematic flow chart illustrating an example method of determining at least one mask for training a mask-based audio processing model based on a DNN according to an embodiment of the present disclosure,

[0065] Figure 7 is a schematic flow chart showing an example method for determining a speech perception loss function according to an embodiment of the present disclosure, the speech perception loss function being used to train a DNN-based audio processing model,

[0066] Figure 8 is a schematic flowchart showing an example method for classifying a target audio signal into a speech frame or a non-speech frame according to an embodiment of the present disclosure, and

[0067] Fig. 9 is a schematic block diagram of an example apparatus for executing a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0068] As noted above, unless otherwise noted, the same or similar reference numerals in the present disclosure may refer to the same or similar elements so that repeated descriptions thereof may be omitted for the sake of brevity.

[0069] In particular, the drawings and the following description relate to the preferred embodiments only by way of illustration. It should be noted that from the following discussion, it will be readily appreciated that alternative embodiments of the structures and methods disclosed herein are feasible alternatives that can be employed without departing from the principles of the claimed invention.

[0070] A detailed description will now be given with reference to several embodiments, examples of which are shown in the accompanying drawings. It should be noted that similar or similar reference numbers may be used in the accompanying drawings where practicable and may represent similar or similar functions. The accompanying drawings describe embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be used without departing from the principles described herein.

[0071] In addition, in the accompanying drawings, connecting elements such as solid lines, dotted lines or arrows are used to illustrate the connection, relationship or association between two or more other schematic elements, and the absence of any such connecting elements does not mean that there is no connection, relationship or association. In other words, the connection, relationship or association between some elements are not shown in the accompanying drawings to avoid blurring the present invention. In addition, for ease of explanation, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, when a connecting element represents the communication of a signal, data or instruction, it should be understood by those skilled in the art that such elements represent (as may be required) one or more signal paths that affect the communication.

[0072] Generally speaking, processing related to speech enhancement is usually aimed at removing one or more unwanted artifacts from a noisy waveform and retaining clean speech. Recently, such speech enhancement processes also tend to utilize machine learning-based techniques, such as deep neural networks (DNNs), etc. It should be noted that although the present disclosure may frequently make references to speech enhancement, etc., it is understood that these are merely possible examples of audio processing in a general sense, and the present disclosure should not be interpreted as being limited to speech enhancement.

[0073] Reference now Figure 1 , which schematically illustrates a schematic diagram of an example audio processing system 100 based on a DNN according to an embodiment of the present disclosure. In particular, a noisy waveform 101 and a far-end echo reference signal 102 can be processed by spectral transform and (optionally) banding 103, and the noisy waveform 101 can be clean speech, noise, echo, reverberation, etc. Subsequently, the DNN 104 can take the mixed band energy as input and try to predict a mask for each time-frequency band 105. The mask value is usually between 0 and 1, which usually represents the (estimated / predicted) presence of speech in the audio mixture. During the training phase, the DNN model may be exposed to a large mixed data set, which can be manually created by randomly adding all signal components (or in a similar way). In some possible implementations, for each sample, an ideal mask can be defined, such as an ideal ratio mask (IRM) 106 according to the above formula (1). The DNN weights will then be updated in a direction that minimizes the loss function 107, which typically measures the difference between the predicted mask and the ideal mask, as will be understood and appreciated by the skilled person.

[0074] However, as briefly discussed above, such systems may suffer from several potential problems in certain circumstances.

[0075] First, it is generally considered that collecting a large amount of clean speech signals of studio quality can be quite challenging. Many times, the "clean" signal (before adding additional noise for the purpose of training for the corresponding audio processing task) may have a certain degree of stationary noise, such as recording noise, room noise, etc. Therefore, conventional IRM metrics are often larger than they should be, and models trained with such IRMs usually cannot remove the associated stationary noise.

[0076] Secondly, in a clean speech signal, there may often be some (certain) time-frequency bands whose energy is relatively small (e.g., below a certain threshold). Specifically, omitting these bands may not be perceived by the end user. With this in mind, in some possible cases, it may be safe to simply set the IRM values ​​of these bands to 0. For the case of DNN-based training, it is generally considered beneficial to set the IRM value to 0, as this appears to help reduce the variability of the target, making it easier for the model to focus on other more meaningful speech bands. However, an algorithm is needed to carefully select the threshold to avoid zeroing out meaningful speech.

[0077] Third, in some possible cases, a speech enhancement DNN may also need to learn (at least implicitly) two behaviors. That is, in speech regions, the DNN may need to try to preserve speech and remove noise, generally preferring to preserve speech. On the other hand, in non-speech regions, the DNN may need to try to aggressively remove unwanted artifacts without worrying about over-suppressing speech. Therefore, it can be seen that the DNN model intrinsically learns the behavior of a voice activity detector (VAD). Therefore, in some possible cases, it may be necessary to include a hand-crafted VAD in the loss function to facilitate learning about when to change the model's aggressiveness (speech frames vs. non-speech frames).

[0078] With this in mind, the present disclosure generally seeks, in a broad sense, to present techniques / mechanisms for performing mask thresholding and / or voice activity detection for the purpose of training a DNN-based speech enhancement model.

[0079] Before describing in detail possible example embodiments of the present disclosure, it may be worthwhile to briefly introduce the per-channel energy normalization (PCEN) technique, which can be understood as an audio signal processing technique used to normalize the energy of each frequency channel in a spectrogram. The technique can be used in a variety of fields and / or applications, such as processing front-end features for robust keyword localization. Broadly speaking, PCEN-based techniques can seek to normalize the energy of each frequency channel in a spectrogram by dividing the energy of the channel by an amount proportional to the average energy of the channel over time. This helps to reduce the effects of background noise and signal level variations across different frequency channels.

[0080] More specifically, in some possible implementations, given a frequency band energy spectrogram S(t, f) of a speech signal, independently for each frequency band f (considered as a function of t), the PCEN metric can be calculated according to the following formula:

[0081]

[0082] where ε = 10 -6 , α=0.98, δ=2 and r=0.5 (or any other suitable values) are (predetermined) constants, and the running average M(t, f) of S(r, f) can be calculated according to the following formula:

[0083] M(t,f)=(1-s)·M(t-1,f)+s·S(t,f) (3)

[0084] Where s is the smoothing factor, which can be set to 0.2 or something similar.

[0085] Of course, it should be noted that the exemplary calculation of the PCEN metric as defined in the above formulas (2) and (3) is provided only as an illustrative example and should not be considered as any form of limitation. As can be understood and appreciated by the skilled person, any other suitable PCEN metric may also be implemented and adopted.

[0086] In addition, it should be noted that the algorithms proposed throughout this disclosure can be applied to both band energy (e.g., as shown above), or, if banding is not used in certain possible cases, binned energy, etc. In addition, the algorithms proposed throughout this disclosure can also be applied to various suitable transforms (e.g., Fast Fourier Transform (FFT), Modulated Discrete Transform (MDXT), Complex Quadrature Mirror Filter (CQMF), etc.) and banding types.

[0087] Now, with reference to the accompanying drawings, the techniques / mechanisms proposed in the present disclosure will be discussed in more detail, which may include (but not limited to) PCEN-based mask thresholding and voice activity detection to improve the loss function of masks (e.g., IRM, etc.) and mask-based speech enhancement model training.

[0088] First, the techniques related to PCEN-based mask thresholding will be discussed. In particular, these techniques are proposed mainly to address the first and second problems of the conventional techniques discussed above. As described in the first problem above, the "clean" speech signal S(t,f) in the above formula (sometimes also understood as the reference truth value for training) may often be accompanied by some stationary noise (e.g., from the recording device, the recording environment, etc.), which may, in some cases, result in the DNN trained using the IRM defined in the above formula (1) being unable to completely remove such stationary noise.

[0089] However, it is generally understood that for each frequency band f (when viewed as a function of t), the speech component may often change much faster than the stationary noise component. Therefore, in the calculation of the PCEN metric for the target speech S(t,f) for each frequency band f (e.g., according to the above formulas (2) and (3)), the denominator term M(t,f) can be viewed as functioning as a low-pass filter due to its smoothing effect.

[0090] In addition, if we change the The term is rewritten as exp(log(S(t,f))-α·log(ε+M(t,f))), then PCEN can be regarded as a high-pass filter in the logarithmic energy domain, which generally subtracts the slowly varying components M(t,f), and in each frequency band f, it is likely that M(t,f) mainly contains stationary noise components.

[0091] Figure 2 A comparison between example frequency band energies 210 of a speech signal with stationary noise and its PCEN-based version 220 is schematically shown.

[0092] In particular, Figure 2 In the example of FIG. 2 , the energy of the frequency band before PCEN, as shown in the upper half of FIG. 210 , appears to contain a low level of stationary noise (e.g., between about 15 and 20 seconds). In contrast, as shown in the lower half of FIG. 220 , after PCEN processing, the stationary noise (between 15 and 20 seconds) appears to be more or less (completely / entirely) removed.

[0093] It should be noted that the effect of high-pass filtering can be understood as being largely controlled by the smoothing factor s in the above formula (3). Specifically, a smaller s may result in more noise removal, but may also severely blur the speech harmonics. Therefore, depending on different implementations and / or applications, it may be necessary to carefully select an appropriate smoothing factor s (e.g., based on experiments or simulations). For example, in some possible cases, it can be observed empirically that s=0.2 (rather than 0.025 in conventional techniques) produces better results in some applications.

[0094] While careful selection of the smoothing factor s may be necessary in some cases, it is generally understood that PCEN may still result in speech loss in some cases (e.g., Figure 2 As schematically shown in the lower half of Figure 220).

[0095] Therefore, it is important to note that the techniques proposed in the present disclosure do not use the PCEN processed speech to define the ideal mask (or in other words, simply replace S(t, f) in formula (1) with the calculated PCEN version). Instead, the proposed techniques generally use the PCEN processed speech signal to determine the threshold. Specifically, if the calculated PCEN(t, f) of S(t, f) is less than the threshold, one option may be to simply and directly set IRM(t, f) to 0; otherwise, the IRM metric may remain the same as defined in formula (1).

[0096] By setting the IRM of frequency bands with stationary noise to 0, the proposed model can be trained to learn to completely remove such stationary noise, thereby helping to solve the first problem discussed above (i.e., involving possible stationary noise hidden in the "clean" signal).

[0097] In addition, as described in the second question above, omitting speech bands with small energy by setting their corresponding IRMs to 0 can help reduce the variability of the training target, thereby allowing the model to focus on meaningful speech bands.

[0098] However, as in Figure 2 As schematically shown in the upper half of Figure 210, the speech energy distribution as a function of (t, f) may vary greatly (especially as a function of f in some possible cases, since low-frequency bands are often considered to be much louder than high-frequency bands). Therefore, using conventional simple methods (such as global methods or band-based mean methods) to determine the threshold without first normalizing the loudness of S(t, f) may result in excessive suppression of speech (especially in high-frequency bands).

[0099] In contrast, since the M(t, f) term in the PCEN calculation is usually a smoothed version of S(t, f), the division in the above formula (2) can normalize the loudness of each frequency band. Figure 2 As shown in the lower half of Figure 220, the energy dynamic range after PCEN appears to be much narrower, and each frequency band appears to have a somewhat similar level. Therefore, the global constant can be used as a threshold - for example, by generating a histogram of the energy of the voice band processed by PCEN for many times, and choosing a small boundary value that does not zero the output speech. In some possible implementation settings, the threshold TH can be used band =10-5 (or similar). Of course, as will be understood and appreciated by a skilled person, the threshold may be determined (eg, calculated) in any suitable manner, and any suitable value may be set accordingly, depending on various implementations and / or applications.

[0100] For ease of understanding, the PCEN-based mask thresholding algorithm proposed above can be summarized as follows:

[0101]

[0102] It should be noted that, as will be understood and appreciated by those skilled in the art, the above algorithm steps (including the parameters / constants used therein) are provided for illustrative purposes only and should not be construed as any type of limitation. Of course, depending on various implementations and / or applications, the steps and / or parameters / constants may be appropriately adjusted.

[0103] Next, techniques related to PCEN-based voice activity detection (VAD) will be discussed. In particular, as discussed above with respect to the third question, in some possible cases, it may be desirable to help the model learn different behaviors in speech frames and non-speech frames and / or in voice bands and non-speech bands. Specifically, for speech frames, it may be necessary to design the loss function to give higher priority to retaining speech than to removing noise; whereas in non-speech frames, it may be necessary to suppress noise as actively as possible (there may also be other unwanted artifacts, such as echoes, etc.). Therefore, in order to achieve such speech perception training, VAD and optionally a loss function guided by VAD may be required. As can be understood and appreciated by technicians, any suitable (existing or new) VAD technology may be used and integrated into the loss implementation of speech perception. For example, an illustrative (non-limiting) example of existing VAD technology may be traditional WebRTC VAD technology.

[0104] In some possible implementations, a simple PCEN-based VAD can be built on top of the band-level PCEN thresholds proposed above. That is, given a target speech S(t, f), for each frame t, the system can be configured to first calculate the frame energy, for example, in the form of the sum of the band energies in the frame. Then, in some possible examples, the system can be configured to empirically set the speech frame-level / non-speech frame-level (or frame-wise) thresholds to:

[0105] TH frame ≈TH band ·N (4)

[0106] where TH bandcan be the band-level PCEN speech threshold previously defined, and N can be the number of bands in each frame. For example, in some possible implementations, N=56 bands in our system can be multiplied by TH band =10 -5 To set TH frame =0.0005. Of course, similar to the above PCEN-based thresholding, it should also be noted that the exemplary calculation of the frame-level threshold defined in the above formula (4) is only provided as an illustrative example and should not be considered as any type of limitation. As can be understood and appreciated by the skilled person, any other suitable implementation may also be adopted.

[0107] Accordingly, if the frame energy is lower than the frame-by-frame threshold, the frame can be empirically classified as a non-speech frame; therefore, the IRM values ​​of all frequency bands in the frame can be set to 0, which can be understood as being able to further reduce the variability of the training target.

[0108] In addition, in some possible implementations, the loss function used to train the DNN model can be designed so that the penalty for excessive suppression of speech is greater than the penalty for insufficient noise removal. As an illustrative (non-limiting) example, the loss function (defined for each time-frequency band, but omitted in the following formula) can be defined as:

[0109] loss = a diff -diff-1 (5)

[0110] In some possible cases, diff can be understood as a mask used to separate the IRM and the estimated mask from the DNN. est The difference between is expressed as:

[0111]

[0112] Where a=2.7 and γ=0.5 (or any other suitable values) can be determined empirically.

[0113] In some possible implementations, both the IRM and the estimated masks proposed in equations (5) and (6) above can be converted to a perceptual scale using a power law; thus, the loss can be viewed as an exponential function of their difference.

[0114] Figure 3 A diagram 300 of an example implementation that plots the loss as a function of diff is schematically shown. In general, the loss function in the example implementation shown is designed to penalize over-suppression of speech more than under-suppression of noise by having a larger gradient value.

[0115] In particular, in example graph 300, the loss function is steeper on the positive side (i.e., where diff>0) corresponding to the model over-suppressing speech (because the IRM is greater than the mask predicted according to equation (6) above), and has a larger gradient (larger slope) than the loss function on the negative side (i.e., where diff<0) where under-suppression occurs where noise occurs. Since techniques related to gradient descent are typically used to minimize the loss by back-propagating the gradient through all model weights, the model weights will be updated faster from the over-suppressed side to the optimal point than from the other side.

[0116] However, this loss function does not seem to distinguish between speech frames and non-speech frames. More specifically, in non-speech frames (e.g., estimated by PCEN-VAD or any other suitable VAD process), because the IRM is set to 0 as described above, excessive suppression of speech usually does not occur (because the predicted mask is usually implemented by a sigmoid function in the last layer of the model, which is usually greater than 0). Therefore, in some possible implementations, it may be advisable to use a larger gradient to actively suppress noise in these (non-speech) frames. For example, in some possible examples, this larger gradient can be achieved by simply flipping (reversing) the order of the two terms in the diff function defined in the above formula (6):

[0117]

[0118] Because as mentioned above, the IRM metric has been (intentionally) set to 0.

[0119] Because diff nonspeec will always be on the positive side of the loss function (see Figure 3 300 in FIG), the (new) loss function will have a larger gradient than the original diff in formula (6), while the diff in formula (6) is negative and therefore has a smaller gradient.

[0120] Figure 4 Schematically shown is a graph 400 that plots the gradient of the proposed loss function with respect to the prediction mask for non-speech frames (where IRM is set to 0 by the PCEN-VAD process described above), from which it can be seen that simply flipping the sign of diff results in a larger slope (compared to Figure 3 Thus, the predicted mask will be driven aggressively towards 0 to remove unwanted artifacts in non-speech frames.

[0121] However, it should be noted that, as will be understood and appreciated by those skilled in the art, such a method of flipping the sign of diff is provided only as an illustrative example. Depending on different implementations and / or applications, any other mechanism for increasing the gradient may be considered. For example, different values ​​of a and γ may be used in the above formulas (5)-(7) for the calculation of speech and non-speech regions, respectively. For example, for speech frames, a may be used. speec =2.7 and γ speech = 0.5 or similar values, which can provide a trade-off between speech preservation and noise suppression. On the other hand, for non-speech frames, a nonspeech >2.7 and γ nonspeech <0.5 or similar values, because in the loss function of non-speech frames, larger a and smaller γ can generally provide larger gradients to more aggressively remove noise. Of course, these examples, including the values ​​mentioned, should not be understood as any form of limitation.

[0122] Similar to the above, the proposed algorithm for PCEN-based VAD and speech perception loss function can be summarized as follows:

[0123]

[0124] It is worth mentioning that, as will be understood and appreciated by the skilled person, the above algorithm steps (including the parameters / constants used therein) are provided for illustrative purposes only and should not be construed as any type of limitation. Of course, depending on various implementations and / or applications, the steps and / or parameters / constants may be appropriately adjusted.

[0125] For example, the general idea of ​​the VAD-controlled loss function can be further extended beyond the specific loss function described above. That is, other suitable loss functions can be made more or less aggressive in speech areas versus non-speech areas, for example, based on VAD values. In some possible implementations, the VAD-based loss function can use time information for different processing strategies for pure noise segments. Specifically, usually due to the sparse nature of speech signals, pure noise may also be present in some possible frequency bands even during speech presence segments.

[0126] Therefore, in some embodiments, for example, a sub-band speech presence probability estimate can be used in the loss function during training. The above-mentioned loss function controlled by VAD can then be extended to a loss function controlled by sub-band speech presence probability. In such cases, any suitable (existing or new) speech presence probability (SPP) estimation method or mechanism can be used in the implementation of the loss function. As an illustrative example of this generalization, the sign flip loss definition described in formula (7) above can be applied not only to non-speech frames (e.g., determined by PCEN-VAD), but also to non-speech bands in speech frames (e.g., determined by band-level PCEN thresholds).

[0127] In addition, in some possible implementations, only one of the algorithms / techniques (i.e., PCEN-based mask thresholding, PCEN-based VAD, speech-perceived loss function) may be used in practice, or more algorithms / techniques may be used simultaneously in practice.

[0128] For example, in some possible cases, when all the above-mentioned proposed techniques are used, PCEN mask thresholding can be first run to set the IRM of the non-speech band to 0 (but these frames can be speech frames). Next, PCEN-based VAD can be performed to set the IRM of the non-speech frames to 0. It should be noted that in both algorithms, the input to PCEN is the original clean speech S (t, f). Further, in some possible cases, the same PCEN parameters can be used, and thus the PCEN metric can be calculated only once. Of course, in some other possible cases, different PCEN parameters may also be used for band-level speech / non-speech decisions relative to frame-level decisions. Finally, the loss of speech perception can be minimized by updating the model weights. It can be understood from experience that the best performance will be obtained by using all of the above techniques / algorithms compared to using only one of the techniques / algorithms, resulting in more noise and (especially) echo being removed.

[0129] It should be noted that, in general, any suitable DNN for predicting IRM, regardless of its architecture, can benefit from the algorithms presented throughout this disclosure, which can be used, for example, to improve the implementation of IRMs and loss functions for training such models.

[0130] For example, in some possible implementations, the proposed PCEN-threshold and PCEN-VAD techniques can be extended to training multi-task models. Figure 5 is a schematic diagram showing an example DNN 500 configured for training a multi-task model according to an embodiment of the present disclosure.

[0131] In particular, if Figure 5 As shown, the model 502 can be trained to take the audio mixture 501 as input and predict K groups of IRMs 503-1, 503-2...503-K. For example, one IRM (e.g., 503-1) can be defined to remove any artifacts (e.g., noise, reverberation, echo, etc.), another IRM (e.g., 503-2) may not remove reverberation (in which case S(t, f) will be reverberant speech), and a third IRM (e.g., 503-K) may be targeted to retain music or speech (in which case S(t, f) will be music or speech). Therefore, one or more PCEN thresholds and PCEN-VAD can be applied to one or more groups of IRMs. That is, PCEN-based techniques can (but do not have to) be applied to some (or all) of the IRM tasks.

[0132] As mentioned above, the algorithms proposed throughout this disclosure can be applied to both band energy (e.g., as shown above), or if banding is not used in some possible cases, it can also be applied to binned energy, etc. In addition, the algorithms proposed throughout this disclosure can also be applied to various suitable transforms (e.g., MDXT, CQMF, etc.) and banding types.

[0133] Figure 6 6 is a schematic flow chart showing a method 600 of determining at least one mask (e.g., one or more masks) according to an embodiment of the present disclosure, the at least one mask being used to train a mask-based audio processing model based on a DNN. As previously mentioned, the model can be trained to perform any suitable audio-related processing, such as (but certainly not limited to) speech enhancement (e.g., denoising, removing artifacts, etc.), etc. Also as can be understood and appreciated by the skilled person, depending on various implementations and / or applications, regardless of the actual architecture, any suitable DNN (or more generally, neural network) implementation can always be used.

[0134] In particular, if Figure 6The illustrated method 600 may start from step S610 by obtaining a time-frequency representation of a target audio signal for training. In general, and particularly in the technical context of the present disclosure, unless otherwise specified, the term "target audio signal" is generally used to refer to an audio signal for which a DNN-based model is trained to perform corresponding audio processing. As an illustrative (non-limiting) example that is easy to understand, in some cases where the model is trained to retain only clean audio signals from a noisy audio mixture (e.g., by removing all unwanted artifacts), the target audio signal may be a clean audio (e.g., speech) signal (e.g., collected by a professional). In some other cases where the model is trained to not remove reverberation (but remove all other artifacts) from a noisy audio mixture, the target audio signal may be a reverberant audio signal (i.e., having reverberation).

[0135] Subsequently, at step S620, method 600 may include determining a PCEN metric for the target audio signal. As will be discussed in more detail below, PCEN may be viewed as an audio signal processing technique for normalizing the energy of each frequency channel in a spectrogram. Broadly speaking, PCEN-based techniques may seek to normalize the energy of each frequency channel in a spectrogram by dividing the energy of the channel by an amount proportional to the average energy of the channel over time. This helps reduce the effects of background noise and signal level variations across different frequency channels.

[0136] Therefore, the method 600 may further include, at step S630 , determining a mask based on the PCEN metric.

[0137] Configured in the manner described above, the proposed method can generally provide an efficient, flexible but reliable mechanism for determining masks that may be used later to train DNN-based audio processing (e.g., speech enhancement) models. Specifically, as described above, the "clean" speech signals collected for training (e.g., S(t, f) in the above formula (1)) often have a certain degree of stationary noise, which may cause the DNN trained using conventional IRM to be unable to successfully and completely remove such stationary noise. Now, in particular, by introducing the PCEN metric as proposed above into the process, (one or more) masks can be determined in a way that can also largely (in some cases even completely) remove such stationary noise (usually embedded in the "clean" signal), thereby significantly improving the performance of the subsequent training process, and thus, significantly improving the final performance in real practice (i.e., after the model has been trained). Incidentally, it should also be noted that, compared with conventional techniques, the PCEN metric proposed in the present disclosure is usually only used in the training phase and not in testing.

[0138] Figure 7is a schematic flow chart illustrating an example method 700 for determining a speech perception loss function for training a DNN-based audio processing model according to an embodiment of the present disclosure.

[0139] In particular, if Figure 7 The method 700 shown can start from step S710 by obtaining the time-frequency representation of the target audio signal for training. Subsequently, in step S720, the method 700 may include determining the presence of speech in the audio signal by using a VAD process. As the technician can understand and recognize, any suitable (existing or new) VAD technology can be used. Finally, the method 700 can also further include determining the loss function based on the presence of speech in the determined target audio signal at step S730. In particular, as the technician can understand and recognize, the loss is usually minimized by back-propagating the gradient in all model weights to use gradient descent. Therefore, based on the presence of speech in the determined target audio signal, the determination of the loss function can be achieved by controlling the gradient of the loss function.

[0140] Configured in the above manner, the proposed method can generally provide an efficient, flexible but reliable mechanism to determine a loss function, which can be used to train a DNN-based audio processing (e.g., speech enhancement) model later. Specifically, according to the above-proposed technique, the loss function can be determined in a design such that (in speech frames) a higher priority is assigned to retaining speech than to removing noise; while in non-speech frames, noise (and other unwanted artifacts, such as echoes, etc.) can be suppressed as aggressively as possible. Therefore, the performance of the training process can be significantly improved, and thus the final performance can also be significantly improved in real practice (i.e., after the model is trained).

[0141] Figure 8 is a schematic flowchart illustrating an example method 800 for classifying a target audio signal into a speech frame or a non-speech frame according to an embodiment of the present disclosure.

[0142] In particular, if Figure 8 The method 800 shown in can be started at step S810 in the following manner: for each audio frame of the target audio signal, a corresponding frame-by-frame PCEN metric is determined across all frequency bands of the audio frame. Subsequently, at step S820, the method 800 may include: if the determined frame-by-frame PCEN metric is greater than a predetermined threshold, the audio frame is classified as a speech frame; otherwise, the audio frame is classified as a non-speech frame.

[0143] Finally, the present disclosure also relates to apparatus for performing the methods and techniques described throughout this disclosure. Fig. 9An example of such an apparatus 900 is generally shown. In particular, the apparatus 900 includes a processor 910 and a memory 920 coupled to the processor 910. The memory 920 may store instructions for the processor 910. Depending on various use cases and / or implementations, the processor 910 may also receive (but not limited to) suitable input data 930 (e.g., an audio signal). Depending on various use cases and / or implementations, the processor 910 may be adapted to perform the methods / techniques described throughout this disclosure (see above). Figures 6 to 8 Any of the methods 600, 700, 800 described in the foregoing) and is suitable for generating corresponding output data 940 (e.g., a mask, a loss function, etc.).

[0144] explain

[0145] A computing device implementing the above techniques may have the following example architecture. Other architectures, including architectures with more or fewer components, are also possible. In some embodiments, the example architecture includes one or more processors (e.g., dual-core The components may include a processor, one or more output devices (e.g., LCD), one or more network interfaces, one or more input devices (e.g., mouse, keyboard, touch-sensitive display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components may communicate and exchange data through one or more communication channels (e.g., buses), where various hardware and software may be used to facilitate the transmission of data and control signals between components.

[0146] The term "computer-readable medium" refers to media that participate in providing instructions to a processor for execution, including but not limited to: non-volatile media (such as optical or magnetic disks), volatile media (such as memory), and transmission media. Transmission media include but are not limited to coaxial cables, copper wire, and optical fiber.

[0147] The computer-readable medium may also include an operating system (e.g., An operating system may include an audio interface manager, an audio processing manager, and a real-time content distributor. The operating system may be multi-user, multi-processing, multi-tasking, multi-threaded, and real-time. The operating system performs basic tasks including, but not limited to: identifying input from and providing output to network interfaces and / or devices; keeping track of and managing files and directories on computer-readable media (such as memory or storage devices); controlling peripheral devices; and managing traffic on one or more communication channels. The network communication module includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols such as TCP / IP, HTTP, etc.).

[0148] The architecture may be implemented in a parallel processing or peer-to-peer infrastructure, or may be implemented on a single device having one or more processors.The software may include multiple software components, or may be a monolithic code.

[0149] The features may be advantageously implemented in one or more computer programs executed on a programmable system, wherein the programmable system includes at least one programmable processor coupled to receive data and instructions from and send data and instructions to a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used in a computer, directly or indirectly, to perform certain activities or cause certain results. Computer programs may be written in any form of programming language (e.g., Objective-C, Java) including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0150] Suitable processors for executing a program of instructions include, for example, both general-purpose and special-purpose microprocessors, and the sole processor of any type of computer or one of multiple cores or multiple processors. In general, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. In general, a computer will also include one or more mass storage devices for storing data files; such devices include disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM optical disks. The processor and memory may be supplemented or included by ASICs (application-specific integrated circuits).

[0151] To provide interaction with a user, features may be implemented on a computer having a display device, such as a retina display device or a CRT (cathode ray tube) or LCD (liquid crystal display) display for displaying information to the user. The computer may have a touch surface input device (e.g., a touch screen) or a keyboard and pointing device such as a mouse or trackball, through which a user can provide input to the computer. The computer may have a voice input device for receiving voice commands from the user.

[0152] Features may be implemented in a computer system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server or an Internet server, or includes a front-end component, such as a client computer with a graphical user interface or an Internet browser, or any combination thereof. The components of the system may be connected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include, for example, a local area network, a wide area network, and the computers and networks that make up the Internet.

[0153] The computing system may include a client and a server. The client and the server are generally remote from each other and generally interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., for displaying data to a user interacting with the client device and receiving user input from it). Data generated on the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0154] A system of one or more computers may be configured to perform an operation by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform a specific operation when run. One or more computer programs may be configured to perform an operation by including instructions that, when executed by a data processing device, cause the device to perform a specific operation.

[0155] Although this specification contains many specific implementation details, they should not be interpreted as limitations on the claims or the scope of any invention, but rather as descriptions of features for specific embodiments of specific inventions. In this specification, certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the individual features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable secondary combination. In addition, although features may be described in the above as working in certain combinations, and even declared as such at the beginning, in some cases, one or more features in the declared combination may be deleted from the combination, and the declared combination may refer to a secondary combination or a variation of a secondary combination.

[0156] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring the operations shown to be performed in the particular order shown or in a sequential manner or requiring all of the operations shown to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of different system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0157] Unless otherwise specifically stated, it will be apparent from the following discussion that discussions throughout the present invention using terms such as "processing," "computing," "calculating," "determining," "analyzing," etc., relate to the operation and / or processing of a computer or computing system, or a similar electronic computing device, that manipulates and / or converts data represented by physical quantities (such as electronically) into other data similarly represented by physical quantities.

[0158] References throughout this disclosure to "an example embodiment," "some example embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in association with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, the phrases "in one example embodiment," "in some example embodiments," or "in example embodiments" appearing throughout this disclosure do not necessarily all refer to the same example embodiment. Furthermore, as will be apparent to one of ordinary skill in the art from this disclosure, particular features, structures, or characteristics may be combined in any suitable manner.

[0159] As used herein, unless otherwise indicated, the use of ordinal adjectives "first," "second," "third," etc. to describe a common object merely indicates that different instances of similar objects are involved, and is not meant to imply that the objects described thereby must be in a given order in time, space, ranking, or in any other manner.

[0160] At the same time, it should be understood that the words and terms used herein are for descriptive purposes and should not be considered limiting. The use of "including," "comprising," "having," and variations thereof are intended to encompass the items listed thereafter and their equivalents as well as additional items. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled," and variations thereof, are used broadly to include direct and indirect mountings, connections, supports, and couplings.

[0161] In the following claims and the description herein, any of the terms "include", "comprises", or "it includes" is an open term that means including at least the elements / features that follow but does not exclude other elements. Therefore, when used in the claims, the term "includes" should not be interpreted as being limited to the means or elements or steps listed thereafter. For example, the expression scope of "a device including A and B" should not be limited to a device consisting of only elements A and B. As used herein, any of the terms "includes", "it includes", or "it contains" is also an open term that means including at least the elements / features that follow the term but does not exclude other terms. Therefore, "include" is synonymous with "include".

[0162] It should be understood that in the above description of example embodiments of the invention, different features of the invention are sometimes combined in a single example embodiment, figure, or description thereof to simplify the invention and aid in the understanding of one or more different inventive aspects. However, this inventive approach should not be interpreted as reflecting an intention that the claims require more features than are explicitly mentioned in each claim. On the contrary, as reflected in the claims below, inventive aspects lie in less than all features of a single above-described example embodiment. Therefore, the claims following the specification are hereby expressly incorporated into this specification in the form of each claim alone as a separate example embodiment of the invention.

[0163] In addition, as will be appreciated by those skilled in the art, although some of the example embodiments described herein include some features included in other example embodiments but do not include other features therein, the combination of features of different example embodiments is also intended to fall within the scope of the present invention and form different example embodiments. For example, in the claims below, any of the claimed example embodiments may be used in any combination.

[0164] In the description provided herein, many specific details are set forth. However, it is understood that the exemplary embodiments of the present invention can be practiced without these specific details. On the other hand, well-known methods, structures and techniques are not described in detail to avoid obscuring the understanding of this specification.

[0165] Therefore, although it is believed that the best mode of the present invention has been described, it will be appreciated by those skilled in the art that other and further modifications may be made without departing from the spirit of the present invention, and the present invention is intended to declare all changes or modifications that fall within the scope of the present invention. For example, any formula given above merely represents a process that may be used. Functions may be added or deleted in the block diagram, and operations may also be exchanged between functional blocks. Steps may be added or deleted in the method described within the scope of the present disclosure.

[0166] The enumerated example embodiment ("EEE") of the present disclosure has been described above with respect to a method and system for determining an indication of audio quality of an audio input. Therefore, embodiments of the present invention may relate to one or more examples enumerated as follows:

[0167] EEE 1. Training a speech enhancement network using targeted PCEN-based mask thresholding.

[0168] EEE 2. Use VAD (such as the proposed PCEN-VAD) to train a speech enhancement network to control the aggressiveness of the loss function.

[0169] EEE 3. A specific speech-aware loss function that incorporates VAD information and controls the aggressiveness of the loss in speech / non-speech frames by flipping the sign of the predicted error.

[0170] EEE 4. A method for processing audio, comprising:

[0171] The neural network is trained using target-oriented channel-wise energy normalization (PCEN) based mask thresholding.

[0172] Use voice activity detection (VAD) to control the aggressiveness of the neural network’s loss function;

[0173] determining a loss function for the neural network based on an ideal ratio mask (IRM) and an estimated mask from the neural network; and

[0174] Provides trained neural networks for speech enhancement.

[0175] EEE 5. The method according to EEE 4, wherein the VAD includes at least one of a frame-level decision or a frequency band-based probability of speech presence. ...

[0176] EEE 6. A method according to any one of EEE 4 or 5, comprising training a multi-task model to predict multiple IRMs, each IRM corresponding to a respective aspect of speech enhancement.

[0177] EEE 7. The method according to any one of EEE 4 to 6, wherein the plurality of IRMs comprises at least one of the following:

[0178] a first IRM configured to remove artifacts including reverberation;

[0179] a second IRM configured to remove artifacts other than reverberation; or

[0180] The third IRM is configured to keep music and voice.

[0181] EEE 8. A method for processing audio, comprising:

[0182] Training a neural network using per-channel energy normalization (PCEN) based mask thresholding for a target; and

[0183] Provides trained neural networks for speech enhancement.

[0184] EEE 9. The method according to any one of EEE 4 to 8, wherein training the neural network comprises:

[0185] for each time-frequency band, calculating a corresponding ideal ratio mask (IRM) based on the time-frequency band energy for clean speech, the time-frequency band energy for noise, and the time-frequency band energy for echo;

[0186] For each time-frequency band, computing a corresponding PCEN filter based on the time-frequency band energy for clean speech;

[0187] For each time-frequency band, if the corresponding PCEN filter for that time-frequency band meets the PCEN threshold, the corresponding IRM is adjusted.

[0188] EEE 10. A method for processing audio, comprising:

[0189] training a neural network, including using voice activity detection (VAD) to control the aggressiveness of a loss function of the neural network; and

[0190] Provides trained neural networks for speech enhancement.

[0191] EEE 11. The method according to any one of EEE 4 to 10, wherein the VAD is a channel-by-channel energy normalization (PCEN) based VAD, wherein the PCEN based VAD operates by performing the following steps:

[0192] For each frame, calculating a corresponding frame energy based on the speech energy across a frequency band of each frame; and

[0193] Each frame is classified as speech or non-speech based on whether the corresponding frame energy meets the PCEN speech threshold.

[0194] EEE 12. The method according to any one of EEEs 4 to 11, wherein the loss function controlled by the VAD comprises a loss function controlled by a sub-band speech presence probability. EEE 13. The method according to any one of EEEs 4 to 11, wherein the loss function controlled by the VAD comprises a loss function controlled by a sub-band speech presence probability.

[0195] EEE 13. A method according to any one of items EEE 4 to 12, wherein the loss function comprises a speech presence probability (SPP) estimation function.

[0196] EEE 14. A method for processing audio, comprising:

[0197] determining a loss function for the neural network based on an ideal ratio mask (IRM) and the estimated mask from the neural network, wherein the loss function is formulated differently for speech frames and non-speech frames or speech bands and non-speech bands; and

[0198] Provides trained neural networks for speech enhancement.

[0199] EEE 15. A system comprising one or more processors configured to perform the operations according to any one of EEE 4 to 14. ...

[0200] EEE 16. A computer program product configured to cause one or more processors to perform operations according to any one of EEE 4 to 14. ...

Claims

1. A method for determining at least one mask for training a mask-based audio processing model based on a deep neural network (DNN), the method comprising: obtaining a time-frequency representation of a target audio signal for said training; Determining a channel-by-channel energy normalized PCEN metric of the target audio signal; as well as The at least one mask is determined based on the PCEN metric. The method according to claim 1 , wherein the target audio signal comprises speech and / or music. The method according to claim 1 , wherein the target audio signal is used as a ground truth for the training.

4. The method according to one of the preceding claims, wherein the target audio signal comprises a clean audio component and a stationary noise component such as recording noise and / or room noise.

5. The method according to one of the preceding claims, wherein the PCEN metric is determined based on a ratio of a time-frequency energy metric of the target audio signal to a running average of the time-frequency energy metric of the target audio signal.

6. The method according to one of the preceding claims, wherein the method further comprises: obtaining an audio mix based on the target audio signal, wherein the audio mix includes audio artifacts in addition to the target audio signal; and Wherein determining the at least one mask based on the PCEN metric comprises: determining at least one ideal ratio mask (IRM) metric based on the target audio signal and the audio mixture; as well as The at least one IRM metric is adjusted based on the PCEN metric to obtain the at least one mask. The method of claim 6 , wherein the audio artifact comprises at least one of: noise, echo, or reverberation.

8. The method according to claim 6 or 7, wherein the IRM measure is determined as a ratio of a time-frequency energy measure of the target audio signal to a time-frequency energy measure of the audio mixture including the audio artifact.

9. The method according to any one of claims 6 to 8, wherein the respective values ​​of the PCEN metric and the IRM metric are determined for each time-frequency band of the target audio signal; and in, The adjustment of the IRM metric includes: For each time-frequency band, if the corresponding PCEN metric value is less than a first predetermined threshold, the corresponding IRM metric value of the time-frequency band is set to 0.

10. The method according to one of the preceding claims, wherein the method further comprises: Classifying the target audio signal into a speech frame or a non-speech frame; as well as For each audio frame, the at least one mask is determined further based on the classification of a speech frame or a non-speech frame.

11. The method according to claim 10, wherein for each audio frame of the target audio signal, the classifying the target audio signal as a speech frame or a non-speech frame comprises: determining corresponding frame-by-frame PCEN metrics across all frequency bands of the audio frame; as well as If the determined frame-by-frame PCEN metric is greater than a second predetermined threshold, classifying the audio frame as a speech frame; Otherwise, the audio frame is classified as a non-speech frame.

12. A method according to claim 10 or 11 when dependent on any one of claims 6 to 9, wherein determining the at least one mask based on the classification of speech frames or non-speech frames comprises: The corresponding IRM metric values ​​of all non-speech frames are set to 0.

13. The method according to one of the preceding claims, wherein for each time-frequency band the PCEN metric is determined according to the following formula: Where S(t, f) is the time-frequency energy of the target audio signal, ε, α, δ and r are predetermined constants, and M(t, f) is the running average of S(t, f) and is defined as: M(t, f)=(1-s)·M(t-1, f)+s·S(t, f), where s is a predetermined smoothing factor.

14. The method of one of the preceding claims, wherein the DNN-based mask-based audio processing model is a multi-task model configured to be trained to predict a plurality of masks, each of the plurality of masks corresponding to a respective audio processing aspect, and wherein the method comprises: For each audio processing aspect, a corresponding target audio signal is obtained; For each target audio signal, determining a corresponding PCEN metric; and A corresponding mask is determined based on the PCEN metric.

15. A method for determining a loss function of speech perception, wherein the loss function of speech perception is used to train an audio processing model based on a deep neural network (DNN), the method comprising: obtaining a time-frequency representation of a target audio signal for said training; determining the presence of speech in the target audio signal by using a voice activity detection (VAD) process; as well as The loss function is determined by controlling a gradient of the loss function based on the determined presence of speech in the target audio signal.

16. The method of claim 15, wherein the VAD process comprises: determining a frame-by-frame presence and / or a frequency band-by-frequency presence of speech in the target audio signal; and Wherein controlling the gradient of the loss function based on the determined presence of speech in the target audio signal comprises: For non-speech frames and / or non-voice bands, increasing the corresponding gradient of the loss function makes the audio artifacts more actively suppressed in the non-speech frames and / or non-voice bands.

17. The method of claim 16, wherein for each audio frame of the target audio signal, determining the frame-by-frame presence of speech in the target audio signal comprises: determining a frame-by-frame channel-by-channel energy normalized PCEN metric across all frequency bands of a corresponding audio frame; and If the determined frame-by-frame PCEN metric is greater than a predetermined threshold, then the corresponding audio frame is determined to contain speech.

18. The method of claim 16 or 17, wherein for each speech frame of the target audio signal, determining the frequency band-by-frequency band presence of speech in the target audio signal comprises: determining a corresponding band-by-band PCEN metric for each band of the speech frame; and If the determined frequency band-by-frequency band PCEN metric is less than a second predetermined threshold, the corresponding frequency band of the speech frame is determined as a non-speech band.

19. The method according to any one of claims 15 to 18, wherein the loss function is determined in such a way that a penalty for over-suppression of speech is greater than a penalty for under-suppression of noise in the target audio signal.

20. The method according to any one of claims 15 to 19, wherein for each time-frequency band, the loss function is defined as: loss=a diff -diff-1, Where a is a predetermined constant, and diff indicates the difference between the ideal ratio mask IRM metric determined for the target audio signal and the estimated mask mask predicted by the DNN-based audio processing model est The difference between .

21. The method according to claim 20, wherein: For a speech frame, the diff is defined as: and For non-speech frames, the diff is defined as: Where γ is a predetermined constant.

22. A method according to claim 20 or 21, wherein the IRM metric is a mask determined by using a method according to any one of claims 1 to 14.

23. The method of any one of claims 15 to 22, wherein determining the loss function comprises: For speech frames and non-speech frames, determining corresponding loss functions; and The loss functions of all time-frequency bands are averaged to obtain the final loss function.

24. A method for training an audio processing model based on a deep neural network (DNN), wherein the DNN-based audio processing model is trained based on: By using the mask determined by the method according to any one of claims 1 to 14, and / or The loss function is determined by using the method according to any one of claims 15 to 23.

25. A method for classifying a target audio signal into a speech frame or a non-speech frame, for each audio frame of the target audio signal, the method comprising: Determining corresponding frame-by-frame channel-by-channel energy normalization PCEN metrics across all frequency bands of the audio frame; as well as If the determined frame-by-frame PCEN metric is greater than a predetermined threshold, classifying the audio frame as a speech frame; Otherwise, the audio frame is classified as a non-speech frame.

26. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to perform a method according to any preceding claim.

27. A program comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 25.

28. A computer-readable storage medium storing the program according to claim 27.

Images