Method and apparatus for improving speech based on deep learning

Abstract

A method, apparatus, program, and storage medium for improving a speech signal based on a neural network. The method includes receiving a speech signal. A first set of features is extracted from the speech signal, each feature in the first set of features corresponding to a frequency bin of the speech signal. The first set of features is grouped into a second set of features, each feature in the second set of features corresponding to a frequency band of the speech signal. A bin mask is estimated based on a neural network, the second set of features being the input to the neural network. The bin mask is applied to the speech signal to generate an improved speech signal.

Description

Technical Field

[0001] Cross - reference to related applications This application claims the benefit of priority of PCT Patent Application No. PCT / CN2023 / 087624, filed on April 11, 2023, the entire content of which is incorporated herein by reference.

[0002] Technical Field This disclosure relates to the improvement of degraded speech signals, and more particularly, to a deep - learning - based speech improvement method and device.

Background Art

[0003] Audio signals may undergo a mixture of degradation due to the environment such as noise, echo reverberation, and degradation associated with processing such as compression, transcoding, and further processing steps before being listened to. This can result in a reduced listening experience for the user as the audio quality of the reproduced audio signal is not satisfactory. For example, a telephone conferencing service provider may find that there is significant degradation of the audio quality before the audio signal is received by the telephone conferencing service. For example, conversations on mobile phones often have GSM - encoded voice before being received by a telephone conferencing service provider. <00,00017>

[0004] Thus, audio signals may sometimes be referred to as degraded audio signals or speech signals, and the improvement of such signals can advantageously be performed to reduce noise, reverberation, and codec artifacts to improve the listening experience.

[0005] When speech enhancement is integrated at the endpoint before the audio signal is presented to the user, the device performing the speech enhancement may not have knowledge of the type of degradation in the received speech signal. For example, the speech enhancement method may not have knowledge of any previously applied compression to the speech signal. For this reason, speech enhancement systems with fixed settings may not be suitable for enhancing received speech signals. To improve speech enhancement in these scenarios, neural network-based speech enhancement has gained popularity because neural networks can be trained on speech containing all types of degradation and can therefore provide improved performance in situations where the actual degradation is unknown to the enhancement method.

[0006] Improving speech in real time using neural network-based speech enhancement can be challenging because it can be difficult to balance the quality of the enhancement with the latency introduced by it.

[0007] Therefore, further improvements are needed in this context. [Overview of the project] [Problems that the invention aims to solve]

[0008] In view of the foregoing, this disclosure provides a neural network-based method, apparatus, and program for speech enhancement, as well as a computer-readable storage medium, having the characteristics of each independent claim. [Means for solving the problem]

[0009] According to certain aspects of this disclosure, a neural network-based method for improving speech signals is provided. Speech signals may be received. Furthermore, a first set of features may be extracted from the speech signal, each feature in the first set of features may correspond to a frequency bin of the speech signal. The first set of features may be grouped into a second set of features, each feature in the second set of features may correspond to a frequency band of the speech signal. A bin mask may be estimated based on a neural network, and the second set of features may be the input to the neural network. The bin mask may be applied to the speech signal to generate an improved speech signal.

[0010] By estimating bin masks based on bandwidth features, the complexity of mask estimation is kept low, while still providing accurate mask values ​​for speech enhancement. This enables the implementation of neural network-based speech enhancement in real-time applications.

[0011] In some embodiments, the speech signal may include speech that has been degraded by one or more of the following: noise, reverberation, compression, and decompression.

[0012] In some embodiments, the features in the first set of features may be the complex spectral values ​​of each frequency bin. Therefore, extracting the first set of features from the speech signal may involve transforming the speech signal into the frequency domain to obtain the transformed speech signal. Furthermore, to obtain the first set of features, features may be extracted from the transformed speech signal for each frequency bin in the frequency domain. Transforming the speech signal into the frequency domain may be performed by any one of the following: short-time Fourier transform (STFT), modified discrete cosine transform (MDCT), shifted discrete frequency transform (MDXT), or filter bank-based transform.

[0013] In some embodiments, grouping a first set of features into a second set of features may involve, for each frequency band of the speech signal, combining features from the first set of features corresponding to the frequency bins within that frequency band to obtain the second set of features. Combining features from the first set of features corresponding to the frequency bins within a frequency band to obtain the second set of features may involve weighting the features from the first set of features corresponding to the frequency bins within a frequency band. Thus, some bin features may be more important than others when determining the band features.

[0014] In some embodiments, the width and spacing of the frequency bands of the speech signals may be perceptually motivated. For example, the frequency bands may be equally spaced in Mel frequency.

[0015] In some embodiments, each feature in the second set of features may be one of the following: Mel-frequency band power, Burk-scale band power, logarithmic frequency band power, or equivalent rectangular bandwidth (ERB) band power.

[0016] In some embodiments, the bin mask may include a value indicating the amount of speech present in each frequency bin of the speech signal. The value may be the ratio of speech to speech plus noise. Therefore, if no noise is present in a frequency bin, the value may be 1 or close to 1. Noise can be understood in this context as any degradation of the speech signal.

[0017] In some embodiments, the neural network may be a deep neural network (DNN). The DNN may include an upsampling module for estimating the bin mask. The DNN may also estimate the band mask based on a second set of features, and may estimate the bin mask based on the band mask and the upsampling module. Alternatively, the function of the upsampling module may be understood as upscaling of the second set of features. The upsampling module may include at least one first module, which includes an upsampling CNN layer, followed by a batch norm layer, followed by an activation layer. The upsampling CNN layer may be a transposed CNN layer. The upsampling module may include multiple consecutive first modules. By using multiple consecutive first modules, the upscaling effect can be adjusted. Thus, the number of consecutive first modules may depend on the ratio of the number of frequency bins to the number of frequency bands.

[0018] In some embodiments, the DNN may further include a feature extraction module, followed by an encoder module, followed by a decoder module, and a CNN layer, where the decoder module is followed by an upsampling module, and the upsampling module is followed by a CNN layer. The encoder module may include at least one downsampling layer and multiple CNN layers, and the decoder module may include at least one upsampling layer and multiple CNN layers.

[0019] In some embodiments, the neural network may be trained on pairs of clean signals and corresponding degraded signals. The training of the neural network may be based on a loss function.

[0020] In some embodiments, applying a bin mask to a speech signal to produce an improved speech signal may include applying the bin mask to the transformed speech signal. Applying the bin mask to the transformed speech signal may include multiplying the value of the bin mask with the transformed speech signal for each frequency bin.

[0021] In some embodiments, the converted speech signal may be converted to the time domain after a bin mask is applied to generate an improved speech signal.

[0022] In some embodiments, the speech enhancement method operates on a single frame of the speech signal. Alternatively, the method may operate on multiple frames of the speech signal. The maximum number of frames on which the method operates may be limited by the acceptable delay for the real-time application.

[0023] Another aspect of this disclosure provides a method for training a neural network to improve the speech of an utterance signal. An utterance signal and a corresponding reference utterance signal may be received. Furthermore, a first set of features may be extracted from the utterance signal, each feature in the first set of features may correspond to a frequency bin of the utterance signal. A third set of features may be extracted from the reference utterance signal, each feature in the third set of features may correspond to a frequency bin of the reference utterance signal. The first set of features may be grouped into a second set of features, each feature in the second set of features may correspond to a frequency band of the utterance signal. A bin mask may be estimated based on the neural network, and the second set of features may be the input to the neural network. A fourth set of features may be determined based on the bin mask and the utterance signal, each feature in the fourth set of features may correspond to a frequency bin of the utterance signal to which the bin mask is applied. A loss function may be evaluated based on the third and fourth sets of features. The parameters of the neural network may be updated based on the evaluated loss function value.

[0024] In some embodiments, the speech signal may be based on a reference speech signal including speech, and the speech signal may be generated by degrading the reference speech signal by one or more of noise, reverberation, compression, and decompression.

[0025] In some embodiments, the features in the first set of features and the third set of features may be complex spectral values of each frequency bin. Therefore, extracting the first set of features from the speech signal may include converting the speech signal into the frequency domain to obtain the converted speech signal. Further, for each frequency bin in the frequency domain, features from the converted speech signal may be extracted to obtain the first set of features. Correspondingly, extracting the third set of features from the speech signal may include converting the reference speech signal into the frequency domain to obtain the converted reference speech signal. For each frequency bin in the frequency domain, features from the converted reference speech signal may be extracted to obtain the third set of features. Converting the speech signal or the reference speech signal into the frequency domain may be performed by any one of a short-time Fourier transform (STFT), a modified discrete cosine transform (MDCT), a shifted discrete frequency transform, an MDXT, or a filter bank-based transform.

[0026] In some embodiments, grouping the first set of features into the second set of features may include, for each frequency band of the speech signal, combining the features of the first set of features corresponding to the frequency bins within that frequency band to obtain the second set of features. Combining the features of the first set of features corresponding to the frequency bins within a frequency band to obtain the second set of features may include weighting the features of the first set of features corresponding to the frequency bins within the frequency band. Therefore, some bin features may be more important than other bin features when determining the band features.

[0027] In some embodiments, the width and spacing of the frequency bands of the speech signal may be perceptually motivated. For example, the frequency bands may be equally spaced in mel frequency.

[0028] In some embodiments, each feature in the second set of features can be any one of mel frequency band power, Bark scale band power, logarithmic frequency band power, or equivalent rectangular bandwidth (ERB) band power.

[0029] In some embodiments, the bin mask can include a value indicating the amount of speech present in each frequency bin of the speech signal. The value can be the ratio of speech to speech + noise. Thus, if there is no noise in the frequency bin, the value can be 1 or close to 1. Noise can be understood as any degradation to the speech signal in this context.

[0030] In some embodiments, the neural network can be a deep neural network (DNN). The DNN can include an upsampling module for estimating the bin mask. The DNN can estimate a band mask based on the second set of features, and can also estimate the bin mask based on the band mask and the upsampling module. Alternatively, the function of the upsampling module can be understood as upscaling the second set of features. The upsampling module can include at least a first module including an upsampling CNN layer, followed by a batch normalization layer, followed by an activation layer. The upsampling CNN layer can be a transposed CNN layer. The upsampling module can include a plurality of consecutive first modules. By using a plurality of consecutive first modules, the upscaling effect can be adjusted. Thus, the number of consecutive first modules can depend on the ratio of the number of frequency bins to the number of frequency bands.

[0031] In some embodiments, the DNN may further include a feature extraction module, followed by an encoder module, followed by a decoder module, and a CNN layer, with the decoder module followed by an upsampling module, and the upsampling module followed by a CNN layer. The encoder module may include at least one downsampling layer and a plurality of CNN layers, and the decoder module comprises at least one upsampling layer and a plurality of CNN layers.

[0032] In some embodiments, determining a fourth set of features based on the bin mask and the speech signal may include applying the bin mask to the transformed speech signal and, after the bin mask has been applied, extracting the fourth set of features from the transformed speech signal. Applying the bin mask to the transformed speech signal may include multiplying the value of the bin mask with the transformed speech signal for each frequency bin.

[0033] In some embodiments, the loss function may be based on the difference between a third set of features and a fourth set of features.

[0034] In some embodiments, the loss function may be a perceptual loss function. The perceptual loss function may use a nonlinear function with an asymmetric penalty for over-suppression or under-suppression. The perceptual loss function is

number

number

[0035] Alternatively, the loss function may be an MSE loss function. The MSE loss function is:

number

[0036] Alternatively, the loss function may be a hybrid loss function. The hybrid loss function may be a weighted sum of the perceptual loss function and the MSE loss function. The hybrid loss function is

number

[0037] In some embodiments, updating the parameters of a neural network based on the evaluated loss function value may include updating the weights within the neural network.

[0038] Another aspect of this disclosure provides an alternative method for training a neural network to improve the speech of an utterance signal. An utterance signal and a corresponding reference utterance signal may be received. Furthermore, a first set of features may be extracted from the utterance signal, each feature in the first set of features may relate to the frequency spectrum of the utterance signal. A third set of features may be extracted from the reference utterance signal, each feature in the third set of features may relate to the frequency spectrum of the reference utterance signal. A mask may be estimated based on the neural network, and the first set of features may be the input to the neural network. The third set of features may be determined based on the mask and the utterance signal. A hybrid loss function may be evaluated based on the third and fourth sets of features. The hybrid loss function may be a combination of a perceptual loss function based on the magnitude of the spectrum and an MSE loss function based on the complex spectrum. The parameters of the neural network may be updated based on the evaluated value of the hybrid loss function.

[0039] In some embodiments, the speech signal may be based on a reference speech signal that includes an utterance, and the speech signal may be generated by degrading the reference speech signal with one or more of the following: noise, reverberation, compression, and decompression.

[0040] In some embodiments, the features in the first and second sets of features may be the complex spectral values ​​of each frequency block. Therefore, extracting the first set of features from the speech signal may involve transforming the speech signal into the frequency domain to obtain the transformed speech signal. Furthermore, to obtain the first set of features, features may be extracted from the transformed speech signal for each frequency block in the frequency domain. Correspondingly, extracting the second set of features from the speech signal may involve transforming a reference speech signal into the frequency domain to obtain the transformed reference speech signal. To obtain the second set of features, features may be extracted from the transformed reference speech signal for each frequency block in the frequency domain. A frequency block may be a frequency bin or frequency band. Transforming the speech signal or reference speech signal into the frequency domain may be performed by one of the following: Short-Time Fourier Transform (STFT), Modified Discrete Cosine Transform (MDCT), Shift Discrete Frequency Transform (MDXT), or Filter Bank-Based Transform.

[0041] In some embodiments, when the frequency block is a frequency band, the width and spacing of the frequency band of the speech signal can be perceptually motivated. For example, the frequency band may be equally spaced in Mel frequency.

[0042] In some embodiments, each feature in the first and second sets of features may be one of the following: Mel-frequency band power, Burk-scale band power, logarithmic frequency band power, or equivalent square band (ERB) band power.

[0043] In some embodiments, the mask may include a value indicating the amount of speech present in each frequency block of the speech signal. The value may be the ratio of speech to speech plus noise. Therefore, if no noise is present in a frequency block, the value may be 1 or close to 1. Noise can be understood in this context as any degradation of the speech signal.

[0044] In some embodiments, the neural network may be a deep neural network (DNN). The DNN may include a feature extraction module, followed by an encoder module, followed by a decoder module, and a CNN layer, where the decoder module is followed by an upsampling module, and the upsampling module is followed by a CNN layer. The encoder module may include at least one downsampling layer and multiple CNN layers, and the decoder module may include at least one upsampling layer and multiple CNN layers.

[0045] In some embodiments, determining a third set of features based on a mask and an utterance signal may include applying a mask to the converted utterance signal and, after the mask has been applied, extracting the third set of features from the converted utterance signal. Applying a mask to the converted utterance signal may include multiplying the value of the mask with the converted utterance signal for each frequency block.

[0046] In some embodiments, the loss function may be based on the difference between a second set of features and a third set of features.

[0047] In some embodiments, the perceptual loss function may be a nonlinear function with an asymmetric penalty for over-suppression or under-suppression.

[0048] In some embodiments, the hybrid loss function may be a weighted sum of the perceptual loss function and the MSE loss function.

number

[0049] In some embodiments, updating the parameters of a neural network based on the evaluated value of the hybrid loss function may include updating the weights within the neural network.

[0050] Aspects of this disclosure may be implemented via a device, which may include a processor and memory coupled to the processor. The processor may be adapted to perform the aspects of this disclosure and the methods according to the embodiments.

[0051] Aspects of this disclosure may be implemented through a program. When the instructions of the program are executed by a processor, the processor can perform the aspects and embodiments of this disclosure. A computer-readable storage medium may store the program. Such a computer-readable storage medium may include memory devices such as, but are not limited to, random access memory (RAM) devices and read-only memory (ROM) devices, as described herein. Thus, some inventive aspects of the subject matter described herein may be implemented through one or more computer-readable storage media on which the software is stored.

[0052] It will be understood that the features of the apparatus and the steps of the method can be replaced in many ways. In particular, the details of the disclosed method can be implemented by the corresponding apparatus (or system), and vice versa, as will be understood by those skilled in the art. Furthermore, it will be understood that any of the above descriptions made relating to the method are equally applicable to the corresponding apparatus (or system), and vice versa. [Brief explanation of the drawing]

[0053] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. [Figure 1] An exemplary framework for training a neural network for speech improvement, according to embodiments of this disclosure, is outlined. [Figure 2]A schematic diagram of an exemplary neural network for improving speech production is shown. [Figure 3] An exemplary neural network having an upsampling module for speech enhancement according to embodiments of the present disclosure is schematically shown. [Figure 4] An upsample module according to an embodiment of this disclosure is schematically shown. [Figure 5] Embodiments This flowchart shows an example of a process for training a neural network for speech improvement according to embodiments of the present disclosure. [Figure 6] This disclosure outlines an exemplary framework for using neural networks to improve speech, according to embodiments of this disclosure. [Figure 7] This flowchart shows an example of a process using a neural network to improve speech according to embodiments of the present disclosure. [Figure 8] The embodiments of this disclosure schematically illustrate another exemplary framework for training a neural network for speech improvement. [Figure 9] This flowchart shows another example of the process for training a neural network for speech improvement according to embodiments of the present disclosure. [Modes for carrying out the invention]

[0054] The drawings and the following description relate to preferred embodiments for illustrative purposes only. It should be noted from the following discussion that alternative embodiments of the structures and methods disclosed herein are readily recognizable as promising alternatives that can be adopted without departing from the principles of the claims.

[0055] Speech enhancement aims to remove multiple undesirable artifacts, such as noise, reverberation, and compression, while preserving the original speech. Recently, deep neural networks (DNNs) have been successfully used in speech enhancement, making DNN-based speech enhancement an attractive research area. A commonly used method for DNN-based speech enhancement is time-frequency masking. More specifically, DNN-based models typically take spectral bin features of the speech signal as input and estimate a time-frequency mask that can be applied to the spectral bin features of the speech signal. However, for many applications operating in real time, estimating the mask using a neural network based on spectral bin features can be too time-consuming. For this reason, existing models have used spectral band features instead of spectral bin features as input for DNNs. The DNN then outputs a time-frequency mask that can be applied to the spectral band features of the speech signal. This significantly reduces the complexity of the DNN model and the delays introduced by processing the speech signal.

[0056] The drawback of this approach is that, compared to a mask containing values ​​for each frequency bin, the lower resolution of the mask results in reduced performance in improving speech.

[0057] The present invention aims to improve speech enhancement while keeping the complexity of the enhancement method low so that real-time enhancement is possible. To achieve this objective, a neural network is proposed that estimates a mask at the bin level based on band-level input features. As an example of such a neural network, a band-to-bin based LensNet model is proposed to increase the resolution of the estimated mask while keeping the complexity as low as the LensNet model described in Patent Document 1, which is incorporated herein by reference in its entirety. [Patent Document 1] PCT Publication WO2022 / 094290, titled "Deep Learning-Based Speech Improvement"

[0058] Furthermore, the performance of DNN-based speech enhancement also depends on the loss function used when the DNN was trained. Recently, a perceptually motivated loss function has been proposed in Patent Document 2, which is incorporated herein by reference in its entirety. [Patent Document 2] PCT Publication WO2023 / 278398, titled "Mitigating hyperinhibition for deep learning-based speech improvement."

[0059] However, perceptually motivated loss functions act on the amplitude spectrum instead of the complex spectrum of the speech signal. When minimizing the loss function by implementing a loss function that ignores complex features in the frequency domain, relevant information may be lost. To leverage knowledge of the complex spectral domain, a hybrid loss function is proposed that combines a perceptual loss function in the absolute spectral domain with a mean squared error (MSE) loss function in the complex spectral domain. This allows for the implementation of nonlinear penalties for over- and under-suppression of degradation in the speech signal, while taking complex domain features into account.

[0060] Hereinafter, several embodiments are given in detail. Examples are shown in the accompanying drawings. Note that whenever it is practical, similar or identical reference numerals may be used in the drawings to indicate similar or identical functions. The drawings illustrate embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be used without departing from the principles described herein.

[0061] Band-to-bin based DNN for speech improvement Figure 1 shows an exemplary framework 100 for training a neural network for speech improvement, according to several embodiments.

[0062] Training a neural network is based on pairs of clean / reference speech signals and degraded speech signals. The degraded speech signal can be generated based on the reference speech signal. The generation of the degraded speech signal may be based on artificial degradation of the speech signal, such as adding a noise floor to the speech signal, or / or on actual degradation resulting from the system transmission chain.

[0063] In the case of artificial degradation, degraded audio utterances can be generated from clean audio utterances in a degraded audio generator. The degraded audio may be part of the same device as the speech enhancement device, or it may be a separate device connected to the speech enhancement device by wire or wirelessly.

[0064] A degraded audio generator can be thought of as embodying multiple simulated transcoding chains. The degraded audio generator receives a clean speech signal and outputs one or more degraded speech signals. Advantageously, one clean speech signal can result in multiple clean-to-degraded audio speech pairs, where the input speech signal is part of each pair, and each pair's degraded speech signal contains different types of artifacts.

[0065] Each simulated transcoding chain in a degraded audio generator includes a set of codecs and filters. For example, generating a degraded speech signal may include applying at least one codec (e.g., an audio codec) to a clean speech signal. Generating a degraded speech signal may also include, alternatively or additionally, applying an Intermediate Reference System (IRS) filter to the clean speech signal. Generating a degraded speech signal may also include, alternatively or additionally, applying a low-pass filter to the clean speech signal.

[0066] Below are 11 examples of transcoding chains that have proven advantageous for training the neural networks described herein. Details of the 11 transcoding chains are as follows: (1) Low-pass filter & IRS8 AMR-NB (5.1) G.71 1 VSV l, (2) Low-pass filter & IRS8 AMR-NB (12.20) G.71 1, (3) Low-pass filter & IRS8 G.729 G.729 (delayed by only 12 samples) G.71 1 VSV, (4) Low-pass filter & IRS8 dynamic range compression Opus Narrowband (6 Kbps) G.71 1 VSV, (5) Low-pass filter & IRS8 Opus Narrowband (6 kbps) AMR-NB (6.70) G.71 1 VSV, (6) Low-pass filter & IRS8 Dynamic range compression AMR-NB (6.70) G.71 1 VSV, (7) Low-pass filter & IRS8 AMR-NB (5.1) MNRU G.71 1 VSV (MOS=3.0), (8) Low-pass filter & IRS8 AMR-NB (5.1) MNRU G.71 1 VSV (MOS=2.5), (9) Low-pass filter & IRS8 CVSD dynamic range compression AMR-NB G.71 1 (simulates GSM mobile phone over Bluetooth®) VSV, (10) Low-pass filter & IRS8 iLBC G.71 1 (simulates iLBC SIP track) VSV, (11) Low-pass filter & IRS8 speex G.71 1 (simulates speex SIP track) VSV.

[0067] The degraded speech signals output from the 11 transcoding chains can be further convolved with a narrowband impulse response before being used to train a neural network to simulate reverberation. Dynamic range compression can be performed by any suitable compressor, depending on the context and requirements.

[0068] Regarding actual degradation, the reference speech signal may be a signal recorded under optimal conditions, and the degraded speech signal may be the same signal recorded with a less capable microphone, and may be processed for transmission over a network such as a wireless network. Processing may include compression and decompression. Compression may be lossy compression, that is, compression may remove content of the audio signal that cannot be regenerated by decompression.

[0069] Both the reference utterance and the corresponding degraded utterance are then processed by the feature extraction module 101, respectively. They may be processed sequentially by the same feature extraction module 101, or in parallel by the same or different feature extraction modules 101. Both utterance signals may be processed frame by frame, i.e., framework 100 may operate on a single frame of the input audio signal. The feature extraction module 101 can process the input audio signal to determine features for each frequency bin of the input audio signal. The number of frequency bins may depend on the sampling frequency and frame size.

[0070] To determine the characteristics of both speech signals, the signals may need to be transformed into the frequency domain. Any suitable discrete frequency transform (Fourier transform, wavelet transform, etc.) can be used. Favorable examples include the Short-Time Fourier Transform (SFTF), Modified Discrete Cosine Transform (MDCT), Shift Discrete Frequency Transform, MDXT, and Filter Bank Transform. The reason for using MDXT instead of MDCT or DFT is that it provides both the energy compactification characteristics of MDCT and similar phase information as DFT.

[0071] Feature determination may involve determining the complex spectral value for each frequency bin.

[0072] In the next step, the bin features of the degraded speech signal are processed by the bandwidth module 102. The bandwidth module 102 groups the bin features into bandwidth features. The process of grouping the bin features may be as follows:

[0073] The spectrum can first be divided into several frequency bands. The frequency bands can be determined such that each band contains the same number of bins (e.g., 100, 160, 200, 320 bins). Alternatively, the frequency bands may each contain a different number of bins. For example, the bandwidth and distribution can be motivated by Mel frequency bands, Burke scale frequency bands, or logarithmic frequency bands.

[0074] Next, for each frequency band, the frequency features corresponding to the bins of that frequency band are combined with the features corresponding to that frequency band. The band-dependent features can be another measure of the power or energy of each band.

[0075] In some embodiments, combining bin features with band features may involve weighting the bin features with different weights.

[0076] In the next step, the determined bandwidth features are input into a neural network. The neural network may be a DNN, which is advantageous. A DNN can be any suitable DNN for mask-based speech enhancement, as it can upscale a relatively small number of bandwidth features into a mask with a value for each frequency bin.

[0077] An example of a DNN that can be modified for this process is shown in Figure 2. This figure shows the structure of the LensNet 200 DNN, in which the output mask has the same dimensions as the input features. A brief description of the general structure of LensNet 200 is given. The LensNet 200 model structure includes a feature extraction module 201, an encoder module 202, a decoder module 203, and a final CNN layer 204. The encoder module 202 may have one or more downsampling layers and other CNN layers. The decoder module 203 may have one or more upsampling layers and other CNN layers. The output of the final CNN layer 204 may be one or more masks. The mask may have the same resolution as the input features; that is, if the input features are band features, the output mask may have values ​​corresponding to each frequency band.

[0078] While LensNet works with low-resolution input features, such as bandwidth features, an upsampling module needs to be added to LensNet so that it can still generate a high-resolution mask containing, for example, values ​​corresponding to each frequency bin. An example of LensNet 300 with an upsampling module is shown in Figure 3. Modules 301, 302, 303, and 304 of LensNet 300 may be identical to modules 201, 202, 203, and 204 of LensNet 200, respectively. The upsampling module 305 may be placed after the decoder module 303 and before the final CNN layer 304. Using the structure of LensNet 300, a mask containing values ​​for each frequency bin can be determined simply using bandwidth features as input.

[0079] An exemplary structure of the upsampling module 400 is shown in Figure 4. The upsampling module 400 may also be the upsampling module 305. The upsampling module 400 may have at least a first module having an inverted CNN layer 401_1, followed by a batch normalization layer 402_2, followed by an activation layer 403_2. The inverted CNN layer 401_1 may be a CNN layer common to other upsampling functions. The upsampling module 400 may include a plurality of consecutive first modules n, each having a CNN layer 401_n, a batch normalization layer 402_n, and an activation layer 403_n, where 1 ≤ n ≤ N, and N is the total number of consecutive first modules. N may depend on the ratio of the number of bins to the number of bandwidths. The stride size of each inverted CNN layer 401_n in the first module may also depend on the ratio of the number of bins to the number of bandwidths. For example, if the ratio is 12, i.e., each band contains 12 bins, or the average number of bins for each band is 12, the number of first modules may be 2, the stride size of each inverted CNN layer 401_n may be 3 or 4, and the kernel size of the inverted CNN layer 401_n may be 6 or 8. The upsample module 400 can estimate the bin mask based on the band mask.

[0080] Returning to Figure 1, the neural network 103 outputs a bin mask, which is the input to the enhancement module 104. The bin mask may contain values ​​for each frequency bin of the degraded speech signal. The value may be the ratio of speech to speech + noise. In this context, noise is understood as any degradation that negatively affects the speech signal, and speech is understood as the speech signal without these degradations. The ratio may be determined by considering the power of the speech signal and the noise signal. Thus, if there is no noise in the speech input signal, the value of the ratio is 1, and if the input speech signal has little speech but a lot of degradation, it approaches 0.

[0081] Furthermore, a second input to the enhancement module 104 is a bin feature corresponding to the degraded speech signal. A bin mask is then applied to the bin features to generate enhanced bin features. Applying the bin mask to the bin features may involve multiplying each value of the bin mask by each corresponding bin feature. The bin features may also be the complex values ​​of each frequency bin. Therefore, the enhanced bin features may also correspond to the complex values ​​of each frequency bin.

[0082] The objective of the speech enhancement framework is to bring the output speech signal as close as possible to the reference / clean speech signal; therefore, in the final step, the enhanced bin features output by the enhancement module 104 must be compared with the bin features of the reference / clean speech signal. This comparison is performed by the loss function 105. Any suitable loss function, such as the mean squared error (MSE), can be used to evaluate the performance of the speech enhancement. Advantageously, a hybrid loss function can be used to further improve the performance of the speech enhancement system. Details regarding the hybrid loss function are described in relation to the embodiment corresponding to Figure 8.

[0083] The results of the loss function are evaluated. The evaluation can be performed on the results of the loss function over multiple pairs of reference / clean utterances and degraded utterances, and over multiple frames of each pair. Ideally, these pairs should capture a wide range of utterance diversity, such as gender and age, and a wide range of degradation for each clean utterance sample. In other words, for each sample of clean utterance, multiple samples of degradation for this particular utterance sample may be provided.

[0084] Depending on the evaluation results, the parameters of the neural network may be updated. Updating the parameters may include updating the weights in the neural network. The neural network may be trained until the result of the loss function reaches a threshold or until the result of the loss function no longer decreases substantially.

[0085] Figure 5 is a flowchart of an example of a process 500 for training a neural network for speech improvement according to an embodiment of the present disclosure. Process 500 may correspond to the steps performed according to the training of the speech improvement framework in Figure 1. In some implementations, blocks of process 500 may be executed by a speech improvement device. Alternatively, blocks of process 500 may be executed by another device, and the parameters for the trained neural network are provided to the speech improvement device.

[0086] In S502, process 500 can receive a speech signal and a corresponding reference speech signal. The speech signal may be generated from the reference speech signal by degrading the speech signal.

[0087] In S504, process 500 may extract a first set of features from the speech signal and a third set of features from the reference speech signal, where each feature in the first set of features corresponds to a frequency bin of the speech signal and each feature in the third set of features corresponds to a frequency bin of the reference speech signal. In order to extract features, the speech signal and the reference signal may have to be converted to the frequency domain. The features in the first set of features and the third set of features may correspond to the complex spectral values ​​of the corresponding frequency bins of the speech signal and the reference speech signal, respectively.

[0088] In S506, process 500 may group the first set of features into a second set of features, where each feature in the second set of features corresponds to a frequency band of the speech signal. Grouping the first set of features into a second set of features may correspond to calculating a value for each frequency band based on the value of the frequency bin in each band. Calculating the frequency band values ​​may include calculating the average amplitude value of the frequency bin within each frequency band.

[0089] In S508, process 500 may estimate a bin mask based on a neural network, where the second set of features is the input to the neural network. Estimating the bin mask may involve estimating a band mask, i.e., a mask containing values ​​for each frequency band, and the neural network upscaling the estimated band mask to a bin mask.

[0090] In S510, process 500 can determine a fourth set of features based on the bin mask and the speech signal, where each feature in the fourth set of features corresponds to a frequency bin of the speech signal to which the bin mask is applied. The fourth set of features may be an improved version of the first set of features. Applying the bin mask to the speech signal may involve multiplying each value of the bin mask by the corresponding value in the first set of features.

[0091] In S512, process 500 may evaluate the loss function based on a third set of features and a fourth set of features. The loss function may be a hybrid loss function that combines the MSE with a perceptual loss function. Evaluating the loss function may involve calculating the results of the loss function over multiple frames of utterance-reference utterance pairs and over multiple utterance-reference utterance pairs.

[0092] In S514, process 500 may update the neural network parameters based on the evaluated loss function values. The parameters may also be the weights of the neural network. The parameter updates may also be based on previous results of the loss function. In particular, the parameter updates may be based on the trend of previous results of the loss function.

[0093] After training the speech enhancement framework, it can be used to improve speech. Figure 6 schematically shows an exemplary framework for using a neural network for speech enhancement according to embodiments of this disclosure. Certain modules in Figure 6 may be identical to those in Figure 1. For a detailed description of these modules, see the embodiments corresponding to Figure 1.

[0094] The speech signal may also be the input to the feature extraction module 601. The speech signal may be degraded due to an unsuitable recording device and / or unsuitable conditions for recording the speech signal, such as background noise. The speech signal may be further degraded due to the transmission of the speech signal to the receiving device, for example, by lossy compression or by compression artifacts. The functionality of the feature extraction module 601 may be identical to that of the feature extraction module 101.

[0095] The extraction module 601 can output bin features of the speech signal. The bin features are grouped into band features by the banding module 602. The functionality of the banding module 602 may be identical to that of the banding module 102.

[0096] The bandwidth features are input to the neural network 603. The neural network 603 may have the same structure as the neural network 103, i.e., it may be a DNN, and preferably structured as shown in Figures 3 and 4. The neural network 603 may be a trained version of the neural network 103, i.e., the weights of the neural network 603 are optimized for speech improvement based on training data. The training data may be pairs of clean and degraded speech. The neural network 603 may output a bin mask for improving the speech signal.

[0097] The bin mask and bin features may be inputs to the enhancement module 604. The enhancement module 604 has the same functionality as the enhancement module 104. Therefore, the enhancement module 604 may output enhanced bin features. The enhanced bin features may correspond to an enhanced version of the input speech signal. To generate the enhanced speech signal, the enhancement module 604 may further perform an inverse frequency transformation corresponding to the frequency transformation performed in the feature extraction module 601. The output of the extraction module 601 may then be the enhanced speech signal in the time domain. Framework 600 may act on a single frame of the input speech signal or on multiple consecutive frames simultaneously. The number of consecutive frames may depend on the content of the speech signal. Furthermore, the number of consecutive frames may be selected so that the delay introduced by speech enhancement is not noticeable to the user of the speech enhancement system in a real-time application. The maximum delay for a real-time application may be in the range of 10 to 80 ms. The corresponding number of consecutive frames may be in the range of 2 to 4.

[0098] Figure 7 is a flowchart of an example of a process 700 that uses a neural network for speech improvement according to an embodiment of the present disclosure. Process 700 may correspond to steps performed according to the training of the speech improvement framework in Figure 6. In some implementations, blocks of process 700 may be performed by a playback device. Alternatively, blocks of process 700 may be performed by another device, and the improved speech signal may be provided to the playback device.

[0099] In S702, process 700 may receive a speech signal. The speech signal may be degraded by one of the following: noise, reverberation, or compression.

[0100] In S704, process 700 can extract a first set of features from the speech signal, where each feature in the first set of features corresponds to a frequency bin of the speech signal. In order to extract features, the speech signal may need to be converted to the frequency domain. The features in the first set of features correspond to the complex spectral values ​​of the corresponding frequency bins of the speech signal.

[0101] In S706, process 700 may group the first set of features into a second set of features, where each feature in the second set of features corresponds to a frequency band of the speech signal. Grouping the first set of features into a second set of features may correspond to calculating a value for each frequency band based on the value of the frequency bin in each band. Calculating the frequency band values ​​may involve calculating the average amplitude value of the frequency bin within each frequency band.

[0102] In S708, process 700 may estimate a bin mask based on a neural network, where the second set of features is the input to the neural network. Estimating the bin mask may involve estimating a band mask, i.e., a mask containing values ​​for each frequency band, and the neural network upscaling the estimated band mask to a bin mask. The neural network may be a trained neural network, more specifically a trained DNN. The neural network may be trained on pairs of clean and degraded utterances.

[0103] In S710, process 700 can apply a bin mask to the speech signal to generate an improved speech signal. Applying a bin mask to the speech signal may include multiplying each value of the bin mask by the corresponding value in a first set of features to generate improved bin features. Generating an improved speech signal may further include applying an inverse frequency transform to the improved bin features to generate an improved version of the speech signal in the time domain.

[0104] Hybrid loss function for training speech improvement neural networks Figure 8 shows another exemplary framework 800 for training a neural network to improve speech, according to several embodiments.

[0105] The neural network is trained based on pairs of clean / reference utterance signals and degraded utterance signals. The degraded utterance signals can be generated based on the reference utterance signals. The generation of degraded utterance signals may be similar to the generation of degraded utterances in framework 100. For details on the generation of degraded utterances, please refer to the embodiment corresponding to Figure 1.

[0106] Both the reference utterance and the corresponding degraded utterance are then processed by the feature extraction module 801, respectively. The reference utterance and the corresponding degraded utterance may be processed sequentially by the same extraction module 801, or in parallel by the same or different extraction modules 801. Both utterance signals may be processed frame by frame, i.e., framework 800 may act on a single frame of the input audio signal. The extraction module 801 processes the input audio signal to determine the features of each frequency block of the input audio signal. A frequency block can be a frequency bin, a frequency band, or any other suitable segmentation of the frequency spectrum. For details regarding bin and band features, please refer again to the embodiment corresponding to Figure 1.

[0107] To determine the characteristics of both speech signals, the signals must be transformed into the frequency domain. Any suitable discrete frequency transform (Fourier transform, wavelet transform, etc.) can be used. Favorable examples include the Short-Time Fourier Transform (SFTF), Modified Discrete Cosine Transform (MDCT), Shift Discrete Frequency Transform, MDXT, and Filter Bank Transform. The reason for using MDXT instead of MDCT or DFT is that it provides both the energy compactification characteristics of MDCT and similar phase information as DFT.

[0108] In the next step, the determined bandwidth features are input to the neural network. The neural network may be a DNN, which is advantageous. The DNN can be any suitable DNN for mask-based speech enhancement. Examples of suitable DNNs are shown in Figures 2 and 3. For details on the DNNs in Figures 2 and 3, please refer again to the corresponding embodiments in Figure 1. The choice of DNN may depend on the type of input features, i.e., the frequency block size of each feature.

[0109] The neural network 803 outputs a mask, which is the input to the enhancement module 804. The mask may contain values ​​for each frequency block of the degraded speech signal. The value may be the ratio of speech to speech + noise. In this context, noise is understood as any degradation that negatively affects the speech signal, and speech is understood as the speech signal without these degradations. The ratio may be calculated by considering the power of the speech signal and the noise signal. Therefore, if there is no noise in the speech input signal, the value of the ratio is 1, and if the input speech signal has little speech but a lot of degradation, it approaches 0.

[0110] Furthermore, a second input to the enhancement module 804 is a feature corresponding to the degraded speech signal. Next, a mask is applied to the feature to generate the enhanced feature. Applying the mask to the feature may involve multiplying each value of the mask by each corresponding feature. The features may be the complex values ​​of each frequency block. Thus, the enhanced feature can also correspond to the complex values ​​of each frequency block.

[0111] The goal of the speech enhancement framework is to make the output speech signal as close as possible to the reference / clean speech signal; therefore, in the final step, the enhanced features output by the enhancement module 804 must be compared with the features of the reference / clean speech signal. This comparison is performed by the hybrid loss function 805.

[0112] Classical loss functions based on MSE may not adequately penalize over-suppression of degraded speech signals. To mitigate over-suppression in neural network-based speech enhancement frameworks, a perceptual relevance cost function is proposed in Patent Document 2. The idea behind the perceptual loss function is to use a nonlinear function with asymmetric penalties for over-suppression or under-suppression. If the difference between the enhanced features and the features of the reference speech signal indicates the presence of an over-suppression event, the loss function sets a greater penalty weight for this situation.

[0113] The proposed perceptual loss function operates in the absolute value spectral domain and does not consider complex domain knowledge. A hybrid loss function is used to evaluate the performance of speech improvement in order to consider not only the absolute value but also the argument of complex spectral values. The hybrid loss function combines the perceptual loss function in the absolute value spectral domain with the MSE in the complex spectral domain.

[0114] In one example, the hybrid loss function is defined as follows:

number

[0115] Based on the experimental evaluation of speech improvement training / inference using band-to-bin LensNet shown in Figure 3, the hybrid loss function exhibits better perceptual quality than the perceptual loss function alone. In other words, when the band-to-bin structure of Figure 1 is used in the framework of Figure 8, performance can be further improved by using the hybrid loss function. However, the hybrid loss function may be used in combination with different configurations.

[0116] The results of the hybrid loss function are evaluated. The evaluation can be performed on the results of the hybrid loss function across multiple pairs of reference / clean utterances and degraded utterances, and across multiple frames within each pair. Ideally, these pairs should capture a wide range of utterance diversity, such as gender and age, and a wide range of degradation for each clean utterance sample. In other words, for each sample of clean utterance, multiple samples of degradation for this particular utterance sample may be provided.

[0117] Depending on the evaluation results, the parameters of the neural network may be updated. Updating the parameters may include updating the weights in the neural network. The neural network may be trained until the result of the hybrid loss function reaches a threshold, or until the result of the hybrid loss function no longer decreases substantially.

[0118] Figure 9 is a flowchart of an example of a process 900 for training a neural network for speech improvement according to an embodiment of the present disclosure. Process 900 may correspond to the steps performed according to the training of the speech improvement framework in Figure 8. In some implementations, blocks of process 900 may be executed by a speech improvement device. Alternatively, blocks of process 900 may be executed by another device, and the parameters for the trained neural network are provided to the speech improvement device.

[0119] In S902, process 900 can receive a speech signal and a corresponding reference speech signal. The speech signal may be generated from the reference speech signal by degrading the speech signal.

[0120] In S904, process 900 can extract a first set of features from the speech signal and a second set of features from the reference speech signal, the first set of features relating to the spectrum of the speech signal and the second set of features relating to the spectrum of the reference speech signal. To extract features, the speech signal and the reference signal may need to be converted to the frequency domain. The features in the first set of features and the second set of features may correspond to the complex spectral values ​​of corresponding frequency blocks for the speech signal and the reference speech signal, respectively. A frequency block may be a frequency bin or frequency band.

[0121] In S906, process 900 may estimate a mask based on a neural network, where the first set of features is the input to the neural network. Estimating the mask may involve estimating a mask with values ​​corresponding to each frequency block.

[0122] In S908, process 900 may determine a third set of features based on the mask and the speech signal. Determining the third set of features may include applying the mask to the speech signal. The third set of features may be an improved version of the first set of features. Applying the mask to the speech signal may include multiplying each value of the mask by the corresponding value in the first set of features.

[0123] In S910, process 900 may evaluate a hybrid loss function based on a third set of features and a second set of features, the hybrid loss function combining a perceptual loss function based on the absolute value of the spectrum and an MSE loss function based on the complex spectrum. Evaluating the hybrid loss function may involve calculating the results of the hybrid loss function over multiple frames of utterance-reference utterance pairs and over multiple utterance-reference utterance pairs.

[0124] In S912, process 900 may update the neural network parameters based on the evaluated value of the hybrid loss function. The parameters may be the weights of the neural network. The parameter update may be based on the preceding result of the hybrid loss function. In particular, the parameter update may be based on the trend of the preceding result of the hybrid loss function.

[0125] interpretation A computing device implementing the above-described technology may have the following exemplary architecture. Other architectures are possible, including architectures with more or fewer components. In some implementations, the exemplary architecture includes one or more processors (e.g., dual-core Intel® Xeon® processors), one or more output devices (e.g., LCDs), one or more network interfaces, one or more input devices (e.g., mouse, keyboard, touch-sensitive display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components can communicate and exchange data through one or more communication channels (e.g., buses), and the communication channels can utilize various hardware and software to facilitate the transfer of data and control signals between the components.

[0126] The term “computer-readable medium” refers to, but is not limited to, any medium involved in providing instructions to a processor for execution, including, but not limited to, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include, but are not limited to, coaxial cables, copper wires, and optical fibers.

[0127] Computer-readable media may further include an operating system (e.g., the Linux® operating system), a network communication module, an audio interface manager, an audio processing manager, and a live content distributor. The operating system may be multi-user, multi-processing, multi-tasking, multi-threaded, real-time, etc. The operating system performs basic tasks including, but not limited to, recognizing input from network interfaces and / or devices and providing output to network interfaces and / or devices; tracking and managing files and directories on computer-readable media (e.g., memory or storage devices); controlling peripheral devices; and managing traffic on one or more communication channels. The network communication module includes various components for establishing and maintaining network connectivity (e.g., software for implementing communication protocols such as TCP / IP and HTTP).

[0128] The architecture can be implemented in parallel processing or peer-to-peer infrastructure, or on a single device having one or more processors. The software can include multiple software components or be a single code body.

[0129] The described features may be advantageously implemented in one or more computer programs executable on a programmable system comprising a data storage system, at least one input device, and at least one output device coupled to at least one programmable processor to receive and transmit data and instructions to them. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform some activity or to produce some result. Computer programs can be written in any form of programming language, including compiled or interpreted languages ​​(e.g., Objective-C, Java®), and can be deployed in any form, such as a standalone program or as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0130] Processors suitable for executing instruction programs include, for example, both general-purpose and dedicated microprocessors, and one of the sole processors or multiple processors or cores in any type of computer. Generally, a processor receives instructions and data from read-only memory or random-access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer also includes, or is operablely coupled to, one or more mass storage devices for storing data files, and such devices include magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and all forms of non-volatile memory, including CD-ROMs and DVD-ROM disks. Processors and memory can be complemented by or incorporated into ASICs (Application-Specific Integrated Circuits).

[0131] To provide user interaction, these features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or retinal display device for displaying information to the user. The computer may have a touch surface input device (e.g., a touchscreen) or a keyboard that allows the user to provide input to the computer, and a pointing device such as a mouse or trackball. The computer may have a voice input device for receiving voice commands from the user.

[0132] The features can be implemented in a computer system that includes backend components such as data servers, middleware components such as application servers or internet servers, or frontend components such as client computers with a graphical user interface or internet browser, or any combination thereof. The components of the system can be connected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include, for example, LANs, WANs, and computers and networks that form the internet.

[0133] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (for example, to display data to a user interacting with the client device and to receive user input from such user). Data generated on the client device (e.g., the results of user interaction) may be received by the server from the client device.

[0134] One or more computer systems may be configured to perform a specific action thanks to software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action while it is running. One or more computer programs may be configured to perform a specific action thanks to containing instructions that cause the data processing device to perform an action when executed by the device.

[0135] This specification includes many specific implementation details, which should not be construed as limitations on the scope of any invention or claimable scope, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, features are described above as acting in certain combinations, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from the combination, and the claimed combination may cover subcombinations or variations of subcombinations.

[0136] Similarly, although the operations are shown in a specific order in the drawings, this should not be understood as requiring that such operations be performed in a specific or sequential order, or that all shown operations be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0137] Unless otherwise specified, as will be apparent from the following discussion, any description using terms such as “process,” “calculate,” “calculate,” “determine,” and “analyze” throughout the discussion of this invention will be understood to refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulates and / or transforms data, which is represented as an electronic or other physical quantity, into other data, which is similarly represented as a physical quantity.

[0138] Throughout this invention, any reference to “one exemplary embodiment,” “several exemplary embodiments,” or “a certain exemplary embodiment” means that certain features, structures, or characteristics described in relation to that exemplary embodiment are included in at least one exemplary embodiment of this invention. Thus, where the phrases “in one exemplary embodiment,” “several exemplary embodiments,” or “a certain exemplary embodiment” appear in various places throughout this invention, they do not necessarily all refer to the same exemplary embodiment. Furthermore, certain features, structures, or characteristics may be combined in any suitable manner in one or more exemplary embodiments, as will be apparent to those skilled in the art from this invention.

[0139] Where used herein, unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to describe a common subject merely indicates that different instances of a similar subject are being referred to, and is not intended to imply that the subjects described in this manner must be in a given order, temporally, spatially, in rank, or in any other way.

[0140] Furthermore, it should be understood that the expressions and terms used herein are for illustrative purposes only and should not be considered limiting. The use of “includes,” “equips,” or “has” and their variations is intended to encompass the enumerated items and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “attached,” “connected,” “supported,” and “joined,” and their variations, are used broadly and encompass both direct and indirect attachment, connection, support, and joining.

[0141] In the following claims and description herein, the terms having, consisting of, or including are open terms meaning that at least the elements / features listed are included, but not excluded from others. Thus, the term "equipped with" when used in the claims should not be interpreted as being limited to the enumerated means, elements, or steps. For example, the expression "a device having A and B" should not be limited to a device consisting only of elements A and B. As used herein, the terms "containing," "including," or "inclusion" are open terms meaning that at least the elements / features listed are included, but not excluded from others. Thus, "containing" is synonymous with "equipped" and means "equipped."

[0142] In the above description of exemplary embodiments of the present invention, it should be understood that various features of the present invention may be grouped together in a single exemplary embodiment, figure, or description for the purpose of improving the flow of the invention and aiding in the understanding of one or more of the various aspects of the invention. However, this method of the present invention should not be construed as reflecting an intention that the claims require more features than expressly described in each claim. Rather, as reflected in the following claims, the aspects of the invention are fewer than all the features of a single, aforementioned disclosed exemplary embodiment. Thus, the claims following this specification are expressly incorporated herein, and each claim stands alone as a separate exemplary embodiment of the present invention.

[0143] Furthermore, some exemplary embodiments described herein include some features included in other exemplary embodiments, but do not include other features, and combinations of features of different exemplary embodiments are intended to fall within the scope of the invention and form different exemplary embodiments, as will be understood by those skilled in the art. For example, any of the claimed exemplary embodiments may be used in any combination within the following claims.

[0144] Numerous specific details are provided in the description herein. However, it is understood that exemplary embodiments of the invention may be carried out without these specific details. On the other hand, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0145] Therefore, while what is considered to be the best mode of the present invention has been described, those skilled in the art will recognize that other further modifications can be made without departing from the spirit of the invention, and all such changes and modifications that fall within the scope of the invention are intended to be claimed. For example, the above formulas simply represent procedures that may be used. Functions may be added to or removed from the block diagram, and operations may be interchanged between function blocks. Steps may be added to or removed from the described methods within the scope of this disclosure.

[0146] Bulleted list exemplary embodiments Various aspects and implementations of this disclosure can also be understood from the following enumerated example embodiments (EEEs) which are not part of the claims. [EEE1] A neural network-based method for improving speech signals, the method being: The step of receiving the aforementioned speech signal; A step of extracting a first set of features from the speech signal, wherein each feature in the first set of features corresponds to a frequency bin of the speech signal; The step of grouping the first set of features into a second set of features, wherein each feature in the second set of features corresponds to the frequency band of the speech signal; A step of estimating a bin mask based on a neural network, wherein the second set of features is the input to the neural network; The steps include applying the bin mask to the speech signal to generate an improved speech signal. Methods that include... [EEE2] The method according to EEE1, wherein the speech signal includes speech that has been degraded by one or more of noise, reverberation, compression, and decompression. [EEE3] Extracting the first set of features from the aforementioned speech signal is: The speech signal is converted to the frequency domain, and the converted speech signal is obtained; To obtain the aforementioned first set of features, the process includes extracting features from the converted speech signal for each frequency bin in the frequency domain. The method described in EEE1 or 2. [EEE4] The method according to EEE3, wherein the conversion of the speech signal into the frequency domain is performed by one of the following: Short-Time Fourier Transform, STFT, Modified Discrete Cosine Transform, MDCT, Shift Discrete Frequency Transform, MDXT, or Filter Bank-Based Transform. [EEE5] Grouping the aforementioned first set of features into the aforementioned second set of features is: For each frequency band of the speech signal, the method includes combining the features of the first set of features corresponding to the frequency bins within that frequency band to obtain the second set of features. The method described in any one of EEE1 to EEE4. [EEE6] The method of EEE5, wherein obtaining a second set of features by combining the features of the first set of features corresponding to frequency bins within a frequency band includes weighting the features of the first set of features corresponding to frequency bins within a frequency band. [EEE7] The width and spacing of the frequency band of the speech signal are perceptually motivated, as described in any one of EEE1 to 6. [EEE8] The method according to EEE7, wherein the frequency bands of the speech signals are equally spaced in Mel frequencies. [EEE9] Each feature in the second set of features corresponds to one of Mel-frequency band power, Burk-scale band power, logarithmic frequency band power, or equivalent square bandwidth (ERB) band power, according to the method of any one of EEE1 to 7. [EEE10] The method according to any one of EEE1 to 9, wherein the bin mask includes a value indicating the amount of speech present in each frequency bin of the speech signal. [EEE11] The method according to EEE10, wherein the value indicating the amount of speech present in each frequency bin of the speech signal is the ratio of speech to speech with noise added. [EEE12] The neural network is a deep neural network (DNN), as described in any one of EEE1 to 11. [EEE13] The method according to EEE12, wherein the DNN has an upsampling module for estimating the bin mask. [EEE14] The method according to EEE13, wherein the DNN estimates a bandwidth mask based on the second set of features, and estimates the bin mask based on the bandwidth mask and the upsampling module. [EEE15] The method according to EEE12 or 13, wherein the upsampling module includes at least one first module comprising an upsampling CNN layer, a subsequent batch normalization layer, and a subsequent activation layer. [EEE16] The upsampled CNN layer is a transposed CNN layer, as described in EEE15. [EEE17] The upsampling module is the method according to EEE15 or 16, wherein the upsampling module has a plurality of consecutive first modules. [EEE18] The method according to EEE17, wherein the number of consecutive first modules depends on the ratio of the number of frequency bins to the number of frequency bands. [EEE19] The method according to any one of EEE13 to 18, wherein the DNN further includes a feature extraction module, a subsequent encoder module, a subsequent decoder module, and a CNN layer, the decoder module being followed by the upsampling module, and the upsampling module being followed by the CNN layer. [EEE20] The method according to EEE19, wherein the encoder module includes at least one downsampling layer and a plurality of CNN layers, and the decoder module includes at least one upsampling layer and a plurality of CNN layers. [EEE21] The method according to any one of EEE1 to 20, wherein the neural network is trained on pairs of clean signals and corresponding degraded signals. [EEE22] The neural network is trained based on a loss function, according to the method described in any one of EEE1 to 21. [EEE23] The method according to any one of EEE3 or, by reference to EEE3, EEE4 to 22, wherein applying the bin mask to the speech signal to generate an improved speech signal includes applying the bin mask to the converted speech signal. [EEE24] The method according to EEE23, with reference to EEE10, wherein applying the bin mask to the converted speech signal includes multiplying the value of the bin mask with the converted speech signal for each frequency bin. [EEE25] The method according to EEE23 or 24, wherein the converted speech signal is converted to the time domain after the bin mask is applied to generate the improved speech signal. [EEE26] The method according to any one of EEE1 to 25, wherein the method is performed on the frame of the speech signal. [EEE27] A method for training a neural network to improve speech output, the method being: The stage of receiving the speech signal and the corresponding reference speech signal; A step of extracting a first set of features from the speech signal and a third set of features from the reference speech signal, wherein each feature in the first set of features corresponds to a frequency bin of the speech signal, and each feature in the third set of features corresponds to a frequency bin of the reference speech signal; The step of grouping the first set of features into a second set of features, wherein each feature in the second set of features corresponds to the frequency band of the speech signal; A step of estimating a bin mask based on the neural network, wherein the second set of features is the input to the neural network; A step of determining a fourth set of features based on the bin mask and the speech signal, wherein each feature in the fourth set of features corresponds to a frequency bin of the speech signal to which the bin mask is applied; A step of evaluating the loss function based on the aforementioned third set of features and the aforementioned fourth set of features; A step of updating the parameters of the neural network based on the evaluated value of the loss function. Methods that include... [EEE28] The method according to EEE27, wherein the speech signal is based on the reference speech signal which includes speech, and the speech signal is generated by degrading the reference speech signal by one or more of noise, reverberation, compression, and decompression. [EEE29] Extracting the first set of features from the aforementioned speech signal is: The speech signal is converted to the frequency domain, and the converted speech signal is obtained; To obtain the aforementioned first set of features, the process includes extracting features from the converted speech signal for each frequency bin in the frequency domain, Extracting the third set of features from the aforementioned reference speech signal is: The aforementioned reference utterance signal is converted to the frequency domain, and the converted reference utterance signal is obtained; To obtain the aforementioned third set of features, the process includes extracting features from the converted reference utterance signal for each frequency bin in the frequency domain. The method described in EEE27 or 28. [EEE30] The method according to EEE29, wherein the conversion of the utterance signal or the reference utterance signal into the frequency domain is performed by one of the following: Short-Time Fourier Transform, STFT, Modified Discrete Cosine Transform, MDCT, Shift Discrete Frequency Transform, MDXT, or Filter Bank-Based Transform. [EEE31] Grouping the aforementioned first set of features into the aforementioned second set of features is: For each frequency band of the speech signal, the method includes combining the features of the first set of features corresponding to the frequency bins within that frequency band to obtain the second set of features. The method described in any one of the EEE28 to 30. [EEE32] The method according to EEE31, wherein obtaining a second set of features by combining the features of the first set of features corresponding to frequency bins within a frequency band includes weighting the features of the first set of features corresponding to frequency bins within a frequency band. [EEE33] The width and spacing of the frequency band of the speech signal are perceptually motivated, as described in any one of EEE27 to 32. [EEE34] The method according to EEE33, wherein the frequency bands of the speech signals are equally spaced in Mel frequencies. [EEE35] The method according to any one of EEE27 to 33, wherein each feature in the second set of features corresponds to one of Mel-frequency band power, Burk-scale band power, logarithmic frequency band power, or equivalent square bandwidth (ERB) band power. [EEE36] The method according to any one of EEE27 to 35, wherein the bin mask includes a value indicating the amount of speech present in each frequency bin of the speech signal. [EEE37] The method according to EEE36, wherein the value indicating the amount of speech present in each frequency bin of the speech signal is the ratio of speech to speech with noise added. [EEE38] The neural network is a deep neural network (DNN), as described in EEE27. [EEE39] The method according to EEE38, wherein the DNN includes an upsampling module for estimating the bin mask. [EEE40] The method according to EEE39, wherein the DNN estimates a bandwidth mask based on the second set of features, and estimates the bin mask based on the bandwidth mask and the upsampling module. [EEE41] The method according to EEE39 or 40, wherein the upsampling module includes at least one first module comprising an upsampling CNN layer, a subsequent batch normalization layer, and a subsequent activation layer. [EEE42] The upsampled CNN layer is a transposed CNN layer, as described in EEE41. [EEE43] The upsampling module is the method according to EEE41 or 42, wherein the upsampling module includes a plurality of consecutive first modules. [EEE44] The method according to EEE43, wherein the number of consecutive first modules depends on the ratio of frequency bins to frequency bands. [EEE45] The method according to any one of EEE39 to 44, wherein the DNN further comprises a feature extraction module, a subsequent encoder module, a subsequent decoder module, and a CNN layer, wherein the decoder module is followed by the upsampling module, and the upsampling module is followed by the CNN layer. [EEE46] The method according to EEE45, wherein the encoder module includes at least one downsampling layer and a plurality of CNN layers, and the decoder module includes at least one upsampling layer and a plurality of CNN layers. [EEE47] The method according to any one of EEE29 or, as referenced to EEE29, EEE30 to 46, wherein determining the fourth set of features based on the bin mask and the speech signal comprises applying the bin mask to the converted speech signal and, after the bin mask has been applied, extracting the fourth set of features from the converted speech signal. [EEE48] The method according to EEE47, as referenced to EEE36, wherein applying the bin mask to the converted speech signal includes multiplying the value of the bin mask with the converted speech signal for each frequency bin. [EEE49] The loss function is the method described in any one of EEE27 to 48, based on the difference between the third set of features and the fourth set of features. [EEE50] The loss function is a perceptual loss function, as described in any one of EEE27 to 49. [EEE51] The method according to EEE50, wherein the perceived loss function uses a nonlinear function with an asymmetric penalty for over-suppression or under-suppression. [EEE52] The method according to EEE50 or 51, wherein the third set of features and the fourth set of features are the amplitude spectra of the reference utterance signal and the utterance signal after the bin mask has been applied, respectively. [EEE53] The aforementioned perceptual loss function is

number

number

number

number

number

Claims

1. A neural network-based method for improving speech signals, the method being: The step of receiving the aforementioned speech signal; A step of extracting a first set of features from the speech signal, wherein each feature in the first set of features corresponds to a frequency bin of the speech signal; The step of grouping the first set of features into a second set of features, wherein each feature in the second set of features corresponds to the frequency band of the speech signal; A step of estimating a bin mask based on a neural network, wherein the second set of features is the input to the neural network; The steps include applying the bin mask to the speech signal to generate an improved speech signal. Methods that include...

2. The method according to claim 1, wherein the speech signal includes speech that has been degraded by one or more of noise, reverberation, compression, and decompression.

3. Extracting the first set of features from the aforementioned speech signal is: The aforementioned speech signal is converted to the frequency domain, and the converted speech signal is obtained; To obtain the aforementioned first set of features, the process includes extracting features from the converted speech signal for each frequency bin in the frequency domain. The method according to claim 1 or 2.

4. The method according to claim 3, wherein the conversion of the speech signal into the frequency domain is performed by one of the following: Short-Time Fourier Transform, STFT, Modified Discrete Cosine Transform, MDCT, Shift Discrete Frequency Transform, MDXT, or Filter Bank-Based Transform.

5. Grouping the aforementioned first set of features into the aforementioned second set of features is: For each frequency band of the speech signal, the method includes combining the features of the first set of features corresponding to the frequency bins within that frequency band to obtain the second set of features. The method according to any one of claims 1 to 4.

6. The method of claim 5, wherein obtaining a second set of features by combining the features of the first set of features corresponding to frequency bins within a frequency band comprises weighting the features of the first set of features corresponding to frequency bins within a frequency band.

7. The method according to any one of claims 1 to 6, wherein the width and spacing of the frequency band of the speech signal are perceptually motivated.

8. The method according to claim 7, wherein the frequency bands of the speech signals are equally spaced in Mel frequencies.

9. The method according to any one of claims 1 to 7, wherein each feature in the second set of features corresponds to one of Mel-frequency band power, Burk-scale band power, logarithmic frequency band power, or equivalent square bandwidth (ERB) band power.

10. The method according to any one of claims 1 to 9, wherein the bin mask includes a value indicating the amount of speech present in each frequency bin of the speech signal.

11. The method according to claim 10, wherein the value indicating the amount of speech present in each frequency bin of the speech signal is the ratio of speech to speech with noise added.

12. The method according to any one of claims 1 to 11, wherein the neural network is a deep neural network (DNN).

13. The method according to claim 12, wherein the DNN has an upsampling module for estimating the bin mask.

14. The method according to claim 13, wherein the DNN estimates a bandwidth mask based on the second set of features, and estimates the bin mask based on the bandwidth mask and the upsampling module.

15. The method according to claim 12 or 13, wherein the upsampling module includes at least one first module comprising an upsampling CNN layer, a subsequent batch normalization layer, and a subsequent activation layer.

16. The method according to claim 15, wherein the upsampled CNN layer is a transposed CNN layer.

17. The method according to claim 15 or 16, wherein the upsampling module has a plurality of consecutive first modules.

18. The method according to claim 17, wherein the number of consecutive first modules depends on the ratio of the number of frequency bins to the number of frequency bands.

19. The method according to any one of claims 13 to 18, wherein the DNN further comprises a feature extraction module, a subsequent encoder module, a subsequent decoder module, and a CNN layer, wherein the decoder module is followed by the upsampling module, and the upsampling module is followed by the CNN layer.

20. The method according to claim 19, wherein the encoder module includes at least one downsampling layer and a plurality of CNN layers, and the decoder module includes at least one upsampling layer and a plurality of CNN layers.

21. The method according to any one of claims 1 to 20, wherein the neural network is trained on pairs of clean signals and corresponding degraded signals.

22. The method according to any one of claims 1 to 21, wherein the neural network is trained based on a loss function.

23. The method according to claim 3 or any one of claims 4 to 22 as referenced to claim 3, wherein applying the bin mask to the speech signal to generate an improved speech signal includes applying the bin mask to the converted speech signal.

24. The method according to claim 23, in reference to claim 10, wherein applying the bin mask to the converted speech signal comprises multiplying the value of the bin mask with the converted speech signal for each frequency bin.

25. The method according to claim 23 or 24, wherein the converted speech signal is converted to the time domain after the bin mask is applied to generate the improved speech signal.

26. The method according to any one of claims 1 to 25, wherein the method is performed on the frame of the speech signal.

27. A method for training a neural network to improve speech output, the method being: The stage of receiving the speech signal and the corresponding reference speech signal; A step of extracting a first set of features from the speech signal and a third set of features from the reference speech signal, wherein each feature in the first set of features corresponds to a frequency bin of the speech signal and each feature in the third set of features corresponds to a frequency bin of the reference speech signal; The step of grouping the first set of features into a second set of features, wherein each feature in the second set of features corresponds to the frequency band of the speech signal; A step of estimating a bin mask based on the neural network, wherein the second set of features is the input to the neural network; A step of determining a fourth set of features based on the bin mask and the speech signal, wherein each feature in the fourth set of features corresponds to a frequency bin of the speech signal to which the bin mask is applied; A step of evaluating the loss function based on the aforementioned third set of features and the aforementioned fourth set of features; A step of updating the parameters of the neural network based on the evaluated value of the loss function. Methods that include...

28. The method according to claim 27, wherein the speech signal is based on the reference speech signal which includes speech, and the speech signal is generated by degrading the reference speech signal by one or more of noise, reverberation, compression, and decompression.

29. Extracting the first set of features from the aforementioned speech signal is: The aforementioned speech signal is converted to the frequency domain, and the converted speech signal is obtained; To obtain the aforementioned first set of features, the process includes extracting features from the converted speech signal for each frequency bin in the frequency domain, Extracting the third set of features from the aforementioned reference speech signal is: The aforementioned reference utterance signal is converted to the frequency domain, and the converted reference utterance signal is obtained; To obtain the aforementioned third set of features, the process includes extracting features from the converted reference utterance signal for each frequency bin in the frequency domain. The method according to claim 27 or 28.

30. The method according to claim 29, wherein the conversion of the utterance signal or the reference utterance signal into the frequency domain is performed by one of the following: Short-Time Fourier Transform, STFT, Modified Discrete Cosine Transform, MDCT, Shift Discrete Frequency Transform, MDXT, or Filter Bank-Based Transform.

31. Grouping the aforementioned first set of features into the aforementioned second set of features is: For each frequency band of the speech signal, the method includes combining the features of the first set of features corresponding to the frequency bins within that frequency band to obtain the second set of features. The method according to any one of claims 28 to 30.

32. The method according to claim 31, wherein obtaining a second set of features by combining the features of the first set of features corresponding to frequency bins within a frequency band comprises weighting the features of the first set of features corresponding to frequency bins within a frequency band.

33. The method according to any one of claims 27 to 32, wherein the width and spacing of the frequency band of the speech signal are perceptually motivated.

34. The method according to claim 33, wherein the frequency bands of the speech signals are equally spaced in Mel frequencies.

35. The method according to any one of claims 27 to 33, wherein each feature in the second set of features corresponds to one of Mel-frequency band power, Burk-scale band power, logarithmic frequency band power, or equivalent square bandwidth (ERB) band power.

36. The method according to any one of claims 27 to 35, wherein the bin mask includes a value indicating the amount of speech present in each frequency bin of the speech signal.

37. The method according to claim 36, wherein the value indicating the amount of speech present in each frequency bin of the speech signal is the ratio of speech to speech with noise added.

38. The method according to claim 27, wherein the neural network is a deep neural network (DNN).

39. The method according to claim 38, wherein the DNN includes an upsampling module for estimating the bin mask.

40. The method according to claim 39, wherein the DNN estimates a bandwidth mask based on the second set of features, and estimates the bin mask based on the bandwidth mask and the upsampling module.

41. The method according to claim 39 or 40, wherein the upsampling module includes at least one first module comprising an upsampling CNN layer, a subsequent batch normalization layer, and a subsequent activation layer.

42. The method according to claim 41, wherein the upsampled CNN layer is a transposed CNN layer.

43. The method according to claim 41 or 42, wherein the upsampling module includes a plurality of consecutive first modules.

44. The method according to claim 43, wherein the number of consecutive first modules depends on the ratio of frequency bins to frequency bands.

45. The method according to any one of claims 39 to 44, wherein the DNN further comprises a feature extraction module, a subsequent encoder module, a subsequent decoder module, and a CNN layer, wherein the decoder module is followed by the upsampling module, and the upsampling module is followed by the CNN layer.

46. The method according to claim 45, wherein the encoder module includes at least one downsampling layer and a plurality of CNN layers, and the decoder module includes at least one upsampling layer and a plurality of CNN layers.

47. The method according to claim 29 or any one of claims 30 to 46 as a reference to claim 29, wherein determining the fourth set of features based on the bin mask and the speech signal comprises applying the bin mask to the converted speech signal and, after the bin mask has been applied, extracting the fourth set of features from the converted speech signal.

48. The method according to claim 47, as referenced to claim 36, wherein applying the bin mask to the converted speech signal comprises multiplying the value of the bin mask with the converted speech signal for each frequency bin.

49. The method according to any one of claims 27 to 48, wherein the loss function is based on the difference between the third set of features and the fourth set of features.

50. The method according to any one of claims 27 to 49, wherein the loss function is a perceptual loss function.

51. The method according to claim 50, wherein the perceived loss function uses a nonlinear function having an asymmetric penalty for over-suppression or under-suppression.

52. The method according to claim 50 or 51, wherein the third set of features and the fourth set of features are the amplitude spectra of the reference utterance signal and the utterance signal after the bin mask has been applied, respectively.

53. The aforementioned perceptual loss function is [Math 1] Defined as such, |S| is the amplitude spectrum of the reference utterance signal, [Math 2] The method according to claim 52, wherein is the amplitude spectrum of the speech signal after the bin mask has been applied, and p is a spectral compression factor.

54. The method according to any one of claims 27 to 49, wherein the loss function is an MSE loss function.

55. The method according to claim 54, wherein the third set of features and the fourth set of features are the complex spectra of the reference utterance signal and the utterance signal after the bin mask has been applied, respectively.

56. The aforementioned MSE loss function is, [Math 3] The method according to claim 55, defined as, where S is the spectrum of the reference utterance signal, ^S is the spectrum of the utterance signal after the bin mask has been applied, p is a spectral compression factor, and the operator φ calculates the argument of a complex number.

57. The method according to any one of claims 27 to 49, wherein the loss function is a hybrid loss function.

58. The method according to claim 57, wherein the hybrid loss function is a weighted sum of the perceptual loss function and the MSE loss function.

59. The aforementioned hybrid loss function is [Math 4] The method according to claim 58, where S is the spectrum of the reference utterance signal, ^S is the spectrum of the utterance signal after the bin mask has been applied, p is a spectral compression factor, operator φ calculates the argument of a complex number, and β is a weighting coefficient.

60. The method according to any one of claims 27 to 59, wherein updating the parameters of the neural network based on the evaluated value of the loss function includes updating the weights in the neural network.

61. A method for training a neural network to improve speech output, the method being: The stage of receiving the speech signal and the corresponding reference speech signal; A step of extracting a first set of features from the speech signal and a second set of features from the reference speech signal, wherein the first set of features relates to the spectrum of the speech signal and the second set of features relates to the spectrum of the reference speech signal; A step of estimating a mask based on the neural network, wherein the first set of features is the input to the neural network; A step of determining a third set of features based on the mask and the speech signal; A step of evaluating a hybrid loss function based on the third set of features and the second set of features, wherein the hybrid loss function combines a perceptual loss function based on the absolute value of the spectrum and an MSE loss function based on the complex spectrum; The steps include updating the parameters of the neural network based on the evaluated value of the hybrid loss function, and Methods that include...

62. The method according to claim 61, wherein the speech signal is generated by degrading the reference speech signal, which includes speech, by noise, reverberation, compression, and decompression, based on the reference speech signal which includes speech.

63. Extracting the first set of features from the aforementioned speech signal is: The aforementioned speech signal is converted to the frequency domain, and the converted speech signal is obtained; To obtain the aforementioned first set of features, the process includes extracting features from the converted speech signal for each frequency block in the frequency domain, Extracting the second set of features from the aforementioned reference utterance signal is: The aforementioned reference utterance signal is converted to the frequency domain, and the converted reference utterance signal is obtained; To obtain the aforementioned second set of features, the process includes extracting features from the converted reference utterance signal for each frequency block in the frequency domain, The method according to claim 61 or 62.

64. The method according to claim 63, wherein the frequency block is a frequency bin or frequency band.

65. The method according to claim 63 or 64, wherein the conversion of the utterance signal or the reference utterance signal into the frequency domain is performed by one of the following: Short-Time Fourier Transform, STFT, Modified Discrete Cosine Transform, MDCT, Shift Discrete Frequency Transform, MDXT, or Filter Bank-Based Transform.

66. The method according to claim 64, wherein when the frequency block is the frequency band, the width and spacing of the frequency band of the speech signal are perceptually motivated.

67. The method according to claim 66, wherein the frequency bands of the speech signals are equally spaced in Mel frequencies.

68. The method according to any one of claims 61 to 66, wherein each feature in the first set of features and the second set of features corresponds to one of Mel-frequency band power, Burk-scale band power, logarithmic frequency band power, or equivalent square bandwidth (ERB) band power.

69. The method according to any one of claims 61 to 68, wherein the mask includes a value indicating the amount of speech present in each frequency block of the speech signal.

70. The method according to claim 69, wherein the value indicating the amount of speech present in each frequency block of the speech signal is the ratio of speech to speech with noise added.

71. The method according to any one of claims 61 to 70, wherein the neural network is a deep neural network (DNN).

72. The method according to claim 71, wherein the DNN includes a feature extraction module, followed by an encoder module, followed by a decoder module, and followed by a CNN layer.

73. The method according to claim 72, wherein the encoder module includes at least one downsampling layer and a plurality of CNN layers, and the decoder module includes at least one upsampling layer and a plurality of CNN layers.

74. Determining a third set of features based on the mask and the speech signal is: The process includes applying the mask to the converted speech signal, and then extracting the third set of features from the converted speech signal after the mask has been applied. The method according to claim 63 or any one of claims 64 to 73 as referenced from claim 63.

75. The method according to claim 74, as referenced to claim 68, wherein applying the mask to the converted speech signal includes multiplying the value of the mask with the converted speech signal for each frequency block.

76. The method according to any one of claims 61 to 75, wherein the hybrid loss function is based on the difference between the second set of features and the third set of features.

77. The method according to any one of claims 61 to 76, wherein the perceptual loss function uses a nonlinear function having an asymmetric penalty for over-suppression or under-suppression.

78. The method according to any one of claims 61 to 77, wherein the hybrid loss function is a weighted sum of the perceived loss function and the MSE loss function.

79. The aforementioned hybrid loss function is [Math 5] Defined as follows, S is the spectrum of the reference utterance signal, ^S is the spectrum of the utterance signal after the bin mask has been applied, p is the spectral compression factor, the operator φ calculates the argument of a complex number, and β is the weight factor. The method according to claim 78.

80. The method according to any one of claims 61 to 79, wherein updating the parameters of the neural network based on the evaluated value of the hybrid loss function includes updating the weights in the neural network.

81. An apparatus having a processor and a memory coupled to the processor, wherein the processor is adapted to perform the method described in any one of claims 1 to 80.

82. A program, when executed by a processor, that includes instructions causing the processor to perform the method according to any one of claims 1 to 80.

83. A computer-readable storage medium storing the program described in claim 82.

Images