Speech enhancement method for transient noise suppression

Abstract

The present invention discloses a speech enhancement method suitable for transient noise suppression, comprising the following steps: (1) performing a short-time Fourier transform on speech data in a clean speech data set to obtain a power spectrum; (2) using the power spectrum to train a VAE model weight; (3) performing a short-time Fourier transform on the noisy signal to be enhanced to obtain a time-frequency spectrum and a power spectrum; (4) performing a NMF on the power spectrum to calculate the time-domain index and frequency-domain index of each mode obtained by decomposition, estimating the components belonging to transient noise therein and filtering them out; (5) inputting the power spectrum after filtering out transient noise into the trained VAE model, combining the NMF with the estimated speech power spectrum and noise power spectrum; (6) subjecting the estimated speech power spectrum and noise power spectrum as well as the time-frequency spectrum of the noisy signal to Wiener filtering to obtain the time-frequency spectrum of the enhanced speech. The present invention can effectively suppress speech noise in a scenario where transient noise exists, while relatively completely preserving the speech components.

Description

Technical Field

[0001] The present invention belongs to the technical field of signal processing, and in particular relates to a speech enhancement method suitable for transient noise suppression. Background Art

[0002] Over the past few decades, speech enhancement has become a significant research and engineering problem, with numerous speech enhancement algorithms proposed and widely adopted. These algorithms can be categorized into two main groups based on their underlying principles: traditional speech enhancement algorithms and deep learning-based algorithms. Traditional speech enhancement algorithms use numerical models to describe speech and noise components, quantify noisy speech, and obtain an estimate of the speech. These algorithms do not require prior knowledge of speaker identity or noise type, resulting in stable and interpretable structures but limited ability to handle non-stationary noise. Deep learning-based speech enhancement algorithms, which have gained popularity in recent years, first train model parameters using mixed speech and noise data samples before enhancing noisy speech test data. While these algorithms demonstrate superior performance on datasets with low signal-to-noise ratios (SNRs), their performance declines when applied to real-world scenarios containing unknown noise, indicating insufficient robustness.

[0003] A recently proposed hybrid speech enhancement scheme that combines semi-supervised training and signal statistical models can combine the advantages of traditional signal processing and deep neural networks. Semi-supervised training means that only clean speech data is used during the network training phase to obtain a generative model of speech. During the testing phase, an unsupervised mathematical model is used to describe the noise, and traditional signal processing techniques are used to obtain enhanced speech. The VAE-NMF scheme (Leglaive S, Girin L, Horaud RA variance modeling framework based on variational autoencoders for speech enhancement[C] / / 2018IEEE28th International Workshop on Machine Learning for Signal Processing (MLSP).IEEE:1-6.), which combines a variational autoencoder (VAE) with non-negative matrix factorization (NMF), is an example of a hybrid speech enhancement algorithm. This scheme uses a semi-supervised VAE model to describe the speech power spectrum, parameterizes the noise power spectrum using NMF, and iteratively updates the unknown parameters using the Monte Carlo Expectation-Maximization algorithm (MCEM). This solution demonstrates good speech enhancement performance in various noise scenarios. However, it cannot effectively remove transient noise, instead enhancing both the transient noise and the speech components. Therefore, it is necessary to improve the VAE-NMF speech enhancement framework by adding a transient noise processing component. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a speech enhancement method suitable for transient noise suppression by combining NMF pre-processing with VAE-NMF framework.

[0005] The technical solution adopted in the present invention is:

[0006] Step 1: Perform short-time Fourier transform on the speech data in the clean speech dataset to obtain its power spectrum;

[0007] Step 2: Build a variational autoencoder (VAE) model, which is composed of a cascade of preprocessing layers, encoders, and decoders. Use the power spectrum obtained in step 1 to train the weights of the variational autoencoder (VAE) model.

[0008] Step 3, performing short-time Fourier transform on the noisy signal to be enhanced to obtain its time spectrum and power spectrum;

[0009] Step 4: Perform non-negative matrix decomposition (NMF) on the power spectrum obtained in step 3, calculate the time domain and frequency domain indices of each mode obtained by decomposition, estimate the components belonging to transient noise and filter them out;

[0010] Step 5: Input the power spectrum obtained in step 4 into the VAE-NMF framework consisting of the trained variational autoencoder (VAE) model and non-negative matrix factorization (NMF) to obtain the estimated speech power spectrum and noise power spectrum.

[0011] Step 6: The speech power spectrum and noise power spectrum obtained in step 5 and the time-frequency spectrum of the noisy signal obtained in step 3 are subjected to Wiener filtering to obtain the time-frequency spectrum of the enhanced speech, and an inverse short-time Fourier transform is performed to obtain the time domain signal of the enhanced speech.

[0012] Compared with the existing technology, the method of the present invention analyzes the distribution characteristics of the time-frequency data of each mode obtained by NMF pre-processing, filters out the components belonging to transient noise, and then inputs the processed power spectrum into the VAE-NMF framework for speech enhancement, thereby achieving better noise suppression performance in the presence of transient noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the method processing in the training stage and the enhancement stage of the present invention.

[0014] Figure 2 2 is a network structure diagram of the VAE model used in the embodiment of the present invention.

[0015] Figure 3 It is the change of the coefficient β in the loss function used in the training model stage of the embodiment of the present invention with the training cycle.

[0016] Figure 4 This is a siSDR comparison chart of the existing VAE-NMF method and the improved VAE-NMF method of the present invention for signal enhancement at different signal-to-noise ratios.

[0017] Figure 5 This is a PESQ comparison chart of the existing VAE-NMF method and the improved VAE-NMF method of the present invention for signal enhancement under different signal-to-noise ratios.

[0018] Figure 6 This is a comparison chart of the STOI of the existing VAE-NMF method and the improved VAE-NMF method of the present invention when enhancing signals under different signal-to-noise ratios. DETAILED DESCRIPTION

[0019] The speech enhancement method for transient noise suppression in this embodiment mainly includes the following steps:

[0020] 1. VAE model and its training

[0021] The VAE model primarily consists of a preprocessing layer, an encoder, and a decoder. Both its input and output are vectors of dimension F. When preparing the training dataset, the mean and variance of each band of the logarithmic power spectrum are recorded. The preprocessing layer first calculates the logarithm of the input signal's power spectrum and then uses this mean and variance to normalize the logarithmic power spectrum. The encoder consists of a combination of multiple multi-layer perceptrons (MLPs) with decreasing dimensionality. Each MLP network contains fully connected layers. In addition to the layers directly connected to the latent variables, it also contains hidden layers, which in turn consist of batch normalization (BN) layers and activation functions. The encoder outputs the latent variables as parameters of a Gaussian distribution. The latent variable samples are randomly sampled from this distribution as input to the decoder. The decoder structure is similar to the encoder, with the number of layers being the reverse of the encoder. The MLP network connected to the output logarithmic power spectrum has no hidden layers.

[0022] The input of the VAE model is the power spectrum of the signal Where F is the frequency point number of the short-time Fourier transform result, n is the frame number, Represents the positive real number domain, ⊙ represents the exponentiation of each element of the matrix or vector. After the preprocessing layer, the normalized logarithmic power spectrum is input into the encoder to obtain the latent variable The Gaussian distribution parameters of , specifically the mean and variance Where L is the dimension of the latent variable, is a real number domain. The latent variable samples are extracted from this distribution and input into the decoder to obtain the reconstructed signal logarithmic power spectrum Where f is the frequency band number.

[0023] 1) Model parameter setting

[0024] The audio data is sampled at 16kHz, and a 64ms sine window with 75% overlap is used for the short-time Fourier transform. The resulting signal power spectrum has dimensions F = 513 and the latent variable has dimensions L = 64. The preprocessing layer converts it into a normalized logarithmic power spectrum. In each MLP layer of the encoder, the dimensions of the fully connected layers (Linear) are 512 and 128, and each Linear layer is followed by a batch normalization layer and a rectified linear unit (ReLU) activation function. After sampling, the latent variable sample z is obtained. n The dimensions of each MLP layer in the decoder are 128 and 512. The specific framework of the VAE model is as follows Figure 2 .

[0025] 2) Loss Function

[0026] The loss function used in this invention is a modified form of maximizing the Evidence Lower Bound (ELBO), where the power spectra of the input and output signals are |s fn | 2 and The lower bound of the evidence consists of two parts:

[0027] i) The difference between the speech reconstructed by VAE and the original input speech

[0028]

[0029] Among them D IS (·‖·) is the IS divergence, R is the number of samples drawn from the latent variable probability distribution, is the rth latent variable sample extracted.

[0030] ii) Differences between the estimated distribution of latent variables and the standard Gaussian distribution

[0031]

[0032] The ratio of the two parts of the loss function is a coefficient β. The loss function used in the training process can be expressed as

[0033]

[0034] 3) Training strategy

[0035] The batch size of the data in the training phase is set to 128, and the optimization strategy of the model parameters uses the Adam optimizer with a learning rate of 0.001. The change of the coefficient β adopts the cyclical annealing strategy (Fu H, Li C, Liu X, et al. Cyclical annealing schedule: A simple approach tomitigating kl vanishing[J]. arXiv preprint arXiv: 1903.10145, 2019.): within an annealing cycle, it gradually increases from 0 to the upper limit 1. After reaching the upper limit or the end of the annealing cycle, β no longer increases, so that VAE repeatedly learns the distribution of changes during training, and both parts of the loss function can be well reduced. Set an annealing cycle to include 50 training cycles, a total of 4 annealing cycles. The change of β during training is as follows Figure 3 The maximum number of training epochs is 500. If the validation set loss does not decrease within 10 epochs, training is terminated early.

[0036] 4) Dataset Usage

[0037] The clean speech data used in training the VAE model comes from the TIMIT database, which contains about 5 hours of single-channel, 16kHz sampling frequency clean speech, which is divided into a training set and a validation set in a 4:1 ratio.

[0038] 2. Use NMF to filter out transient noise components

[0039] 1) Perform NMF on the power spectrum of the input noisy signal. The iterative formula is:

[0040]

[0041] in is the power spectrum of the noisy signal, is the frequency domain feature matrix, is the time domain activation matrix, is a matrix of all ones, ⊙ and / represent the multiplication / division of the elements in the corresponding positions of the two matrices, respectively. Where F and N are the number of frequency bands / time frames of the short-time Fourier transform result, and K is the rank of the set non-negative matrix factorization.

[0042] 2) Calculate the time domain index of each mode obtained by NMF. The specific method is: for a row vector h that constitutes H k , first find the location N where its maximum value is max , search h from this position to the left and right sides of the vector k The value shows an inflection point of rising, and the two endpoints of a complete peak are obtained. l and Nr Calculate the time domain index D h,k The formula is

[0043]

[0044] Among them, h kn Represents vector h k The nth element of .

[0045] 3) Calculate the frequency domain indicators of each mode obtained by NMF. The specific method is as follows: for the time domain signal with a sampling frequency of 16kHz, a short-time Fourier transform is performed to obtain a frequency domain signal with a maximum frequency of 8kHz. This continuous frequency segment is divided into three frequency ranges according to the different sound components contained:

[0046] ① 0~40Hz, ultra-low frequency sound, mostly background noise;

[0047] ② 40~3000Hz, the frequency range where human voices mainly appear;

[0048] ③ 3000~8000Hz, high-frequency noise and a small amount of human voice.

[0049] For a column vector w that makes up W k ,use Duan Yu The spectral energy ratio of the segment is used as the frequency domain index D to measure its energy distribution w,k , the calculation formula is

[0050]

[0051] Among them, w fk Represents vector w k The fth element of .

[0052] 4) Based on the calculated time domain and frequency domain indicators of each mode, determine the components belonging to transient noise and remove them:

[0053]

[0054] where θ h and θ w is the lower bound parameter of the time domain and frequency domain indicators, V b is the estimated noise power spectrum, V s is the estimated speech power spectrum, X is the time-frequency spectrum of the noisy speech, and X′ is the time-frequency spectrum of the noisy speech after filtering out the transient noise component.

[0055] 3. VAE-NMF solution

[0056] 1) Signal variance model

[0057] Use the variance model to describe the noisy speech signal after NMF pre-processing:

[0058]

[0059] where x fn , s fn and b fn Represent the time-frequency spectrum of noisy speech, speech and noise obtained by short-time Fourier transform, is the time domain scale parameter. It is assumed that the time-frequency spectra of speech and noise obey a complex Gaussian distribution with a mean of 0 and a variance of the power spectral density:

[0060]

[0061] Use NMF to describe the noise variance v b,fn , which can be expressed as:

[0062] v b,fn =(W b H b ) fn (10)

[0063] Among them, W b and H b The frequency domain feature matrix and time domain activation matrix are obtained by decomposing the noise variance.

[0064] 2) Parameter iterative update

[0065] The parameters to be updated include the latent variable z n , the noise NMF matrix W b and H b , time domain scale specification parameter g. The MH algorithm (Metropolis-Hastings algorithm) (Robert CP, Casella G, Casella G. MonteCarlo statistical methods: vol. 2 [M]. Springer, 1999.) is used to update the latent variables. In each iteration, the MH algorithm uses a random walk strategy to update the latent variables based on the last extraction. Extract new latent variables z from the distribution after adding perturbation ∈ n :

[0066]

[0067] in is the identity matrix, and then calculate the acceptance probability α:

[0068]

[0069] Among them, p(z) and p(z (m-1) ) is the prior probability of the latent variable, p(x|z) and p(x|z (m-1) ) is the conditional probability of the noisy speech with respect to the latent variable calculated according to formula (9). Then α is compared with the sample u randomly drawn from the 0-1 uniform distribution, and the z corresponding to each frame signal is n Independent comparison. If α>u, then update the value of z; otherwise, the value of z remains unchanged.

[0070] Update W according to the EM algorithm b , H b and g, using the block-coordinate method, construct an auxiliary function to update W separately and in sequence b , H b and g, the iterative formulas are:

[0071]

[0072]

[0073]

[0074] in:

[0075]

[0076] After each parameter is updated, the values ​​of the variables affected by the change must be recalculated. For example, when W is updated b After that, you need to update synchronously

[0077] 4. Wiener filtering to reconstruct enhanced signal

[0078] After obtaining the corresponding parameter estimates Finally, the time-frequency spectrum of the enhanced speech is obtained using Wiener filtering:

[0079]

[0080] right Perform inverse short-time Fourier transform to obtain the enhanced speech signal.

[0081] Example

[0082] The technical solutions in the embodiments of the present invention are described clearly and completely below with reference to the accompanying drawings.

[0083] 1. Parameter settings

[0084] In the NMF pre-processing module, the rank of the matrix decomposition is set to 16, the number of iterations is 100, and the lower bound parameter θ is set to h =0.9,θ w = 0.1; in the VAE-NMF model, the rank of the matrix decomposition is 10, and the perturbation variance in the MH sampling algorithm ∈ 2 =0.01, the maximum number of iterations of the MCEM algorithm is 100, and the iteration stops early when the Q value decreases.

[0085] 2. Simulation Example

[0086] 1) Test sample

[0087] The noisy speech used in the test was synthesized from clean speech and noise. The clean speech data came from the TIMIT database test set, which contains 10 sentences spoken by 168 people, with one sentence randomly selected from each speaker. The noisy speech data came from the DEMAND database, from which a noise segment of the same length as the selected sentence was randomly extracted. The speech and noise were superimposed at three signal-to-noise ratios of -5, 0, and 5 to generate the noisy speech test samples.

[0088] 2) Evaluation indicators

[0089] The performance of speech enhancement algorithms is evaluated using three metrics: i) scale-invariant Signal-to-Distortion Ratio (siSDR), measured in decibels (dB); ii) Perceptual Evaluation of Speech Quality (PESQ), ranging from -0.5 to 4.5; and iii) Short-Time Objective Intelligibility (STOI), ranging from 0 to 1. Higher values ​​for all three metrics indicate better speech enhancement.

[0090] 3) Simulation results

[0091] In order to demonstrate the performance of the method of the present invention, this embodiment will be compared with the existing VAE-NMF solution (Leglaive S, Girin L, Horaud RA variance modeling framework based on variational autoencoders for speech enhancement [C] / / 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE: 1-6.). Figure 4 、 Figure 5 and Figure 6 The speech enhancement performance of the existing VAE-NMF scheme (VAE-NMF-ori) and the improved VAE-NMF scheme (VAE-NMF-imp) at three signal-to-noise ratios (SNRs) of -5, 0, and 5 is shown. The values ​​in the figure are the averages of the enhancement results for all test samples under a given metric. It can be seen that the improved VAE-NMF scheme consistently outperforms the existing VAE-NMF scheme across various metrics at different SNRs.

Claims

1. A speech enhancement method for transient noise suppression, characterized in that: The steps include: Step 1: Perform short-time Fourier transform on the speech data in the clean speech dataset to obtain its power spectrum; Step 2: Build a variational autoencoder (VAE) model, which is composed of a cascade of preprocessing layers, encoders, and decoders. Use the power spectrum obtained in step 1 to train the weights of the variational autoencoder (VAE) model. Step 3, performing short-time Fourier transform on the noisy signal to be enhanced to obtain its time spectrum and power spectrum; Step 4: Perform non-negative matrix decomposition (NMF) on the power spectrum obtained in step 3, calculate the time domain and frequency domain indices of each mode obtained by decomposition, estimate the components belonging to transient noise and filter them out; Step 5: Input the power spectrum obtained in step 4 into the VAE-NMF framework consisting of the trained variational autoencoder (VAE) model and non-negative matrix factorization (NMF) to obtain the estimated speech power spectrum and noise power spectrum. Step 6: The speech power spectrum and noise power spectrum obtained in step 5 and the time-frequency spectrum of the noisy signal obtained in step 3 are subjected to Wiener filtering to obtain the time-frequency spectrum of the enhanced speech, and an inverse short-time Fourier transform is performed to obtain the time domain signal of the enhanced speech.

2. The speech enhancement method for transient noise suppression according to claim 1, characterized in that: In step 4, the iterative formula for performing non-negative matrix decomposition (NMF) on the power spectrum obtained in step 3 is: in is the power spectrum of the noisy signal, is the frequency domain feature matrix, is the time domain activation matrix, is a matrix of all ones, ⊙ and / represent the multiplication / division of the elements in the corresponding positions of the two matrices respectively; where F and N are the number of frequency bands / time frames of the short-time Fourier transform results, and K is the rank of the set non-negative matrix decomposition.

3. The speech enhancement method for transient noise suppression according to claim 2, characterized in that: In step 4, the method for calculating the time domain index of each mode obtained by decomposition is: for a row vector h constituting the matrix H, k , first find the location N where its maximum value is max , search h from this position to the left and right sides of the vector k The value shows an inflection point of rising, and the two endpoints of a complete peak are obtained. l and N r ; Calculate the time domain index D h,k The formula is: Among them, h kn Represents vector h k The nth element of .

4. The speech enhancement method for transient noise suppression according to claim 2, characterized in that: In step 4, the method for calculating the frequency domain index of each modal obtained by decomposition is as follows: for the time domain signal with a sampling frequency of 16kHz, a frequency domain signal with a maximum frequency of 8kHz is obtained through short-time Fourier transform; the continuous frequency segment is divided into three frequency ranges according to the different sound components contained: ① 0~40Hz, ultra-low frequency sound, mostly background noise;② 40~3000Hz, the frequency range where human voices mainly appear; ③ 3000~8000Hz, high-frequency noise and a small amount of human voice; For a column vector w that constitutes the matrix W k ,use Duan Yu The energy ratio of the segment is used as a frequency domain indicator to measure its energy distribution. w,k , the calculation formula is: Among them, w fk Represents vector w k The fth element of .

5. The speech enhancement method for transient noise suppression according to claim 1, wherein: In step 4, based on the calculated time domain index and frequency domain index, the mode belonging to transient noise is determined and removed. The specific formula is: where θ h and θ w is the lower bound parameter of the time domain and frequency domain indicators, V b is the estimated noise power spectrum, V s is the estimated speech power spectrum, X is the time-frequency spectrum of the noisy speech, and X′ is the time-frequency spectrum of the noisy speech after filtering out the transient noise component.

Images