Lightweight time-domain convolutional network speech enhancement method and system

Abstract

The present invention provides a lightweight time-domain convolutional network speech enhancement method. First, the noisy speech signal is subjected to frame windowing processing, and then a short-time Fourier transform is performed to obtain the amplitude spectrum and phase spectrum of the noisy speech signal. Subsequently, the amplitude spectrum of the noisy speech signal is sent to a time-domain convolutional network model to estimate the priori signal-to-noise ratio, and then the estimated priori signal-to-noise ratio is used to estimate the gain function. Finally, filtering and noise reduction are performed in the amplitude domain, and the phase spectrum of the noisy speech signal is used as the enhanced phase to reconstruct the speech signal. The present invention also provides a lightweight time-domain convolutional network speech enhancement system. The beneficial effects of the present invention are: the neural network part of the algorithm only estimates the priori signal-to-noise ratio, which reduces the difficulty of the task, realizes a lighter neural network architecture, has a small memory footprint, and has low computational complexity, so that it can be better used on some embedded devices on low-computation platforms.

Description

Technical Field

[0001] The present invention relates to a speech enhancement method, and in particular to a lightweight time-domain convolutional network speech enhancement method and system. Background Art

[0002] At present, voice interaction products on the market, such as smart phones, service robots, voice recorders, translators, etc., all have built-in voice enhancement algorithms to eliminate background noise in the environment, thereby achieving the purpose of improving the user experience in a noisy environment. The latest research shows that deep neural networks have shown excellent results in voice enhancement. However, the noise reduction method based on deep neural networks also has the disadvantages of large memory usage and high computational complexity, making it difficult to use on some embedded devices with low computing platforms. Summary of the invention

[0003] In order to solve the problems in the prior art, the present invention provides a lightweight time-domain convolutional network speech enhancement method and system based on a Bayesian statistical model.

[0004] The present invention provides a lightweight time-domain convolutional network speech enhancement method. First, a noisy speech signal is subjected to frame windowing processing, and then a short-time Fourier transform is performed to obtain an amplitude spectrum and a phase spectrum of the noisy speech signal. Subsequently, the amplitude spectrum of the noisy speech signal is sent to a time-domain convolutional network model to estimate a priori signal-to-noise ratio, and then a gain function is estimated using the estimated priori signal-to-noise ratio. Finally, filtering and denoising are performed in the amplitude domain, and the phase spectrum of the noisy speech signal is used as the enhanced phase to reconstruct the speech signal.

[0005] The reconstruction of speech signals includes inverse Fourier transform and overlap-add.

[0006] As a further improvement of the present invention, the construction process of the time domain convolutional network model is as follows:

[0007] First, according to the definition of the priori signal-to-noise ratio, its value range is any number greater than zero, as shown in formula (1):

[0008]

[0009] Among them, |X clean (n,k)| and |N noise (n,k)| represents the amplitude spectrum of pure speech and noise signal respectively, n and k are frame index and frequency index respectively. In order to reduce the learning difficulty of the time domain convolutional network model, its value range is compressed. Through statistical research on the prior signal-to-noise ratio, it is found that its logarithmic transformation form, as shown in formula (2), obeys Gaussian distribution. Therefore, a cumulative distribution function is used to compress it, as shown in formula (3):

[0010] ξ dB (n,k)=10·log 10 (ξ(n,k)) (2)

[0011]

[0012] Among them, ξ dB (n,k) is the prior signal-to-noise ratio after logarithmic compression, and It is the prior signal-to-noise ratio after compression by the cumulative distribution function, and its value range is [0,1]. erf(·) represents the cumulative distribution function, μ k and σ k are obtained through statistical analysis at different frequencies. dB The mean and variance corresponding to (n,k);

[0013] Next, considering the temporal dependency between speech signals, a lightweight time-domain convolutional network model is designed using dilated convolution to learn the amplitude spectrum of noisy speech |X noisy (n,k)| and the prior signal-to-noise ratio The mapping relationship between them.

[0014] As a further improvement of the present invention, the loss function during the training of the temporal convolutional network model is calculated using cross entropy, and the calculation formula is shown in formula (4), where y i and They are the ideal priori signal-to-noise ratio and the priori signal-to-noise ratio results predicted by the neural network output layer:

[0015]

[0016] As a further improvement of the present invention, the gain function of the noise reduction filter is estimated by using the gain function expression derived from the Bayesian theory and the minimum mean square error criterion, as shown in the following formula (5):

[0017]

[0018] Among them, G(n,k) is the estimated gain function, ξ(n,k) is the estimated value of the prior signal-to-noise ratio, which is also the lower limit of the integral. However, since the prior signal-to-noise ratio output by the temporal convolutional network is a compressed value, it is necessary to perform an inverse transformation on it to obtain the prior signal-to-noise ratio required in equation (5). The inverse transformation process is shown in equations (6) and (7):

[0019]

[0020]

[0021] Substituting the inverse transformed priori signal-to-noise ratio into the gain function expression in equation (5), we can obtain the gain value of the amplitude spectrum of each frame of noisy speech at each frequency point. By performing a dot multiplication with each frequency point, we can obtain the enhanced speech amplitude value. Finally, the phase spectrum of the noisy speech can be used to synthesize the final enhanced speech signal.

[0022] As a further improvement of the present invention, the time domain convolutional network model is composed of two fully connected layers and N residual modules, and the input features are amplitude spectrum features of noisy speech.

[0023] As a further improvement of the present invention, the input features are normalized and then reshaped into a tensor form of [number of sentences, sentence length, 1, 257] for model training or inference, wherein the fourth dimension 257 of the tensor corresponds to the amplitude spectrum features corresponding to each frame of speech data extracted. Subsequently, the transformed input features are obtained through a 256-dimensional fully connected layer. In order to better represent the input features, a nonlinear rectifier unit (ReLU) is used for nonlinear activation. Next, considering the temporal dependency between speech signals, the information between different frames is captured by stacking residual modules with hole convolutions, and the context information in the past frames is used to estimate the prior signal-to-noise ratio of the current frame. In terms of the design of the residual module, each residual module is composed of three convolutional layers. In order to reduce the number of parameters and the amount of computation, the residual module is used. The bottleneck structure is designed in this way. The output channel dimensions of the three convolutional layers are 128, 128, and 256 respectively, and the corresponding tensor sizes are [number of sentences, sentence length, 1, 128], [number of sentences, sentence length, 1, 128], and [number of sentences, sentence length, 1, 256] respectively. In order to further reduce the number of parameters and computational complexity of the model, the first and third convolutional layers are grouped from left to right. The number of groups is set to 8. After the grouping operation, the channel dimension of each group is 16, and then 1×1 convolution is used to concatenate the channel information in each group. In order to ensure information interaction between groups, the output of the first layer of grouped convolution is channel mixed. In each group, the channels corresponding to the same index are arranged from left to right, and then a void convolution operation with a convolution kernel of 1×3 is performed on each channel, and the void rate is 1. Different residual modules are configured in a 2, 4, 8, and 16 loop manner, and the last convolution layer performs the same grouped convolution operation as the first convolution layer, and then adds the input and output features of the residual module through skip connections. At the end of the entire time domain convolutional network model, a 257-dimensional fully connected layer is used to estimate the prior signal-to-noise ratio. Of course, in order to ensure that its output range is between [0,1], the sigmoid function is used as the activation function of the fully connected layer.

[0024] The present invention also provides a lightweight time-domain convolutional network speech enhancement system, comprising a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are used to implement any of the methods described above when executed by a processor.

[0025] The beneficial effects of the present invention are as follows: the neural network part of the algorithm only estimates the prior signal-to-noise ratio, which reduces the difficulty of the task and realizes a lighter neural network architecture with less memory usage and low computational complexity, so that it can be better used on some embedded devices on low-computing platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a lightweight time-domain convolutional network speech enhancement method based on a Bayesian statistical model of the present invention.

[0027] Figure 2 This is a structural diagram of a lightweight time convolutional neural network model of a lightweight time-domain convolutional network speech enhancement method based on a Bayesian statistical model of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0029] A lightweight time-domain convolutional network speech enhancement method based on a Bayesian statistical model aims to combine a deep neural network with a minimum mean square error speech estimator based on a statistical model. It not only utilizes the big data learning ability of the neural network, but also combines the statistical thinking of the minimum mean square error estimator. It can better cope with different noise scenarios and improve the generalization performance of the algorithm.

[0030] The whole algorithm framework mainly consists of two parts, one is the speech enhancement part based on the minimum mean square error criterion, and the other is the estimation part of the prior signal-to-noise ratio, such as Figure 1 shown.

[0031] Figure 1 It is the overall block diagram of the speech enhancement system designed by the present invention. There are two key technologies, one is the estimation of the prior signal-to-noise ratio, and the other is the estimation of the gain function for frequency domain filtering. The noisy speech signal is first framed and windowed, and then short-time Fourier transform is performed to obtain the amplitude spectrum and phase spectrum of the noisy speech signal. Subsequently, the amplitude spectrum of the noisy speech is sent to the time domain convolutional network model to estimate the prior signal-to-noise ratio, and then the gain function is estimated using the estimated prior signal-to-noise ratio. Finally, filtering and noise reduction are performed in the amplitude domain, and the phase of the noisy speech is used as the enhanced phase to reconstruct the speech signal.

[0032] The core parts of the entire algorithm system lie in the design of the time domain convolutional network and the estimation of the gain function, which will be introduced in detail below.

[0033] First, according to the definition of the priori signal-to-noise ratio, its value range is any number greater than zero, as shown in formula (1).

[0034]

[0035] Among them, |X clean (n,k)| and |N noise (n,k)|represents the amplitude spectrum of pure speech and noise signal respectively, n and k are frame index and frequency index respectively. In order to reduce the learning difficulty of the neural network model, its value range is compressed here. Through statistical research on the prior signal-to-noise ratio, we found that its logarithmic transformation form, as shown in formula (2), obeys Gaussian distribution. Therefore, a cumulative distribution function is used here to compress it, as shown in formula (3).

[0036] ξ dB (n,k)=10·log 10 (ξ(n,k)) (2)

[0037]

[0038] Among them, ξ dB (n, k) is the prior signal-to-noise ratio after logarithmic compression, and is the prior signal-to-noise ratio after compression by the cumulative distribution function, and its value range is [0,1]. erf(·) represents the cumulative distribution function, μ k and σ k are obtained through statistical analysis at different frequencies. dB The mean and variance corresponding to (n,k).

[0039] Next, considering the temporal dependency between speech signals, we design a lightweight temporal convolutional network model using dilated convolution to learn the amplitude spectrum of noisy speech |X noisy (n,k)| and the prior signal-to-noise ratio The mapping relationship between them.

[0040] The lightweight time convolutional network structure designed by the present invention is as follows: Figure 2As shown, the entire network structure is composed of two layers of fully connected layers and N residual modules. The input features are the amplitude spectrum features of noisy speech. In order to ensure a more stable convergence process, the input features are normalized and then reshaped into a tensor form of [number of sentences, sentence length, 1, 257] for model training or inference, where the fourth dimension of the tensor 257 corresponds to the amplitude spectrum features corresponding to each frame of speech data extracted. Subsequently, the transformed input features are obtained through a 256-dimensional fully connected layer. In order to better represent the input features, a nonlinear rectifier unit (ReLU) is used here for nonlinear activation. Next, considering the temporal dependency between speech signals, the present invention captures the information between different frames by stacking residual modules with dilated convolutions, and uses the context information in past frames to estimate the prior signal-to-noise ratio of the current frame. In terms of the design of the residual module, each residual module consists of three convolutional layers. In order to reduce the number of parameters and the amount of computation, the present invention adopts a bottleneck structure design method. The output channel dimensions of the three convolutional layers are 128, 128, and 256 respectively, and the corresponding tensor sizes are [number of sentences, sentence length, 1, 128], [number of sentences, sentence length, 1, 128], and [number of sentences, sentence length, 1, 256]. In order to further reduce the number of parameters and computational complexity of the model, the present invention groups the first and third convolutional layers from left to right. Here, the number of groups is set to 8. After the grouping operation, the channel dimension of each group is 16, and then 1×1 convolution is used to concatenate the channel information in each group. In order to ensure information interaction between groups, we mix the channels of the output of the first layer of grouped convolution. In each group, the channels corresponding to the same index are arranged from left to right. Then, a dilated convolution operation with a convolution kernel of 1×3 is performed on each channel. The dilated rate is configured for different residual modules in a cyclic manner of 1, 2, 4, 8, and 16. The last convolution layer performs the same grouped convolution operation as the first layer of convolution, and then adds the input and output features of the residual module through skip connections. At the end of the entire neural network architecture, a 257-dimensional fully connected layer is used to estimate the prior signal-to-noise ratio. Of course, in order to ensure that its output range is between [0,1], the sigmoid function is used as the activation function of the fully connected layer. Through experiments, we found that when the number of residual modules N is 20, the designed temporal convolution network has excellent noise reduction capabilities, and the parameters of the entire neural network model are only 332K, and the number of floating-point operations is 670K, which can fully meet the computing power limitations of most embedded platforms.

[0041] In addition, after the temporal convolutional neural network model is built, it needs to be trained with a large amount of data to enable it to have the ability to map the prior signal-to-noise ratio.

[0042] To train the model, we first need to prepare enough pairs of noisy speech amplitude values ​​and ideal prior signal-to-noise ratio as training data sets, so we chose the TIMIT data set. [1] 4620 sentences are used as the pure speech data of the training set, and then NOISEX-92 [2] The 12 noises in the noise library, including restaurant noise, 2 fighter noises, 2 destroyer noises, factory noise, tank noise, Volvo car noise, high-frequency channel noise, white noise, Panther tank noise and machine gun noise, are randomly mixed with clean speech as noise data. The mixed signal-to-noise ratio is between [-5,15] and obeys uniform distribution. A total of about 38 hours of noisy training data is obtained. In order to tune the parameters of the model, a validation set needs to be set. 280 sentences are also selected from the TMIT dataset as the validation set clean speech data, and are uniformly mixed with the 12 noises in the training set with a signal-to-noise ratio of -5 to 15dB.

[0043] The loss function during training of the temporal convolutional network model is calculated using cross entropy, as shown in equation (4), where y i and They are the ideal priori signal-to-noise ratio and the priori signal-to-noise ratio results predicted by the neural network output layer:

[0044]

[0045] In order to improve the overfitting problem of the model, each layer of the neural network adopts a 20% neuron random inactivation rate and batch normalization regularization method. During training, the Adam optimization algorithm is used for back propagation, iterating 40 times with a learning rate of 0.001, and then iterating 10 times with a learning rate of 0.0001, to obtain a temporal convolutional network model with the ability to map the prior signal-to-noise ratio.

[0046] After obtaining the prior signal-to-noise ratio through the neural network module, the next step is to estimate the gain function of the noise reduction filter. This part mainly uses the gain function expression derived from Bayesian theory and the minimum mean square error criterion. [3] , as shown in the following formula (5).

[0047]

[0048] Among them, G(n,k) is the estimated gain function, ξ(n,k) is the estimated value of the prior signal-to-noise ratio, which is also the lower limit of the integral. However, since the prior signal-to-noise ratio output by the temporal convolutional network is a compressed value, it is necessary to perform an inverse transformation on it to obtain the prior signal-to-noise ratio required in equation (5). The inverse transformation process is shown in equations (6) and (7).

[0049]

[0050]

[0051] Substituting the inverse transformed priori signal-to-noise ratio into the gain function expression in equation (5), we can obtain the gain value of the amplitude spectrum of each frame of noisy speech at each frequency point. By performing a dot multiplication with each frequency point, we can obtain the enhanced speech amplitude value. Finally, the phase spectrum of the noisy speech can be used to synthesize the final enhanced speech signal.

[0052] We also verified the noise reduction effect of the method proposed in the present invention through experiments. In order to evaluate the quality, intelligibility and distortion of the noise-reduced speech, we used PESQ (Perceptual evaluation of speech quality), STOI (Short-Time Objective Intelligibility) and SDR (Signal to Distortion Ratio) indicators to evaluate the noise-reduced speech. As shown in Table 1, all noise reduction effects and indicators are measured on the test set. The higher the indicator, the better the performance. The test set used is another 320 sentences selected from the TIMIT data set that are not repeated in the training set and the validation set, and are mixed with 12 trained noises and 3 untrained noises (untrained fighter noise, untrained factory noise and pink noise) in NOISEX-92 into five noise pollution levels of -5dB, 0dB, 5dB, 10dB and 15dB.

[0053] The experimental results in Table 1 show that the method proposed in the present invention can not only have a good noise reduction effect in the trained noise scene, but also can be well generalized to the untrained noise scene, and has good noise reduction robustness. In addition, when we conducted subjective listening, we found that the speech processed by this method has good intelligibility and quality, and sounds very comfortable. In addition, the instantaneous noise such as the knocking of machines and machine guns in the factory is also suppressed very well, and even in some low signal-to-noise ratio noise environments, there is little residual noise.

[0054] Table 1 Evaluation results of PESQ, STOI and SDR indicators under different noise environments

[0055]

[0056] The present invention provides a lightweight time-domain convolutional network speech enhancement method based on a Bayesian statistical model. It uses the powerful data modeling ability of deep neural networks to learn a more accurate speech priori signal-to-noise ratio estimation model from a large amount of speech data, and combines it with the minimum mean square error speech estimator based on Bayesian theory to realize a lightweight, high-performance noise reduction algorithm. The main advantages of this algorithm are as follows:

[0057] (1) The algorithm realizes the complementary advantages of neural networks and statistical models, and improves the robustness of the algorithm in different noise environments;

[0058] (2) The neural network part of the algorithm only estimates the prior signal-to-noise ratio, which reduces the difficulty of the task and achieves a lighter neural network architecture;

[0059] (3) The algorithm can effectively suppress instantaneous noise;

[0060] (4) The algorithm can achieve better noise reduction effect in low signal-to-noise ratio environment.

[0061] References:

[0062] [1] JS Garofolo, “Getting started with the DARPA TIMIT CD ROM: Anacoustic phonetic continuous speech database NIST Tech Report,” 1988.

[0063] [2] Andrew Varga, Herman JM, Steeneken, "Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems," Speech Communication, vol. 12, no. 3, 1993.

[0064] [3] Y.Ephraim and D.Malah, "Speech enhancement using a minimum mean-square error log-spectral amplitude estimator," IEEE / ACM Transactions onAudio, Speech and Language Processing, vol.33, no.2, pp.443–445, 1985.

[0065] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A lightweight time-domain convolutional network speech enhancement method, characterized by: Firstly, the noisy speech signal is processed by frame division and windowing, and then short-time Fourier transform is performed to obtain the amplitude spectrum and phase spectrum of the noisy speech signal. Then, the amplitude spectrum of the noisy speech signal is sent to the time domain convolutional network model to estimate the prior signal-to-noise ratio. Then, the gain function is estimated using the estimated prior signal-to-noise ratio. Finally, filtering and noise reduction are performed in the amplitude domain, and the phase spectrum of the noisy speech signal is used as the enhanced phase to reconstruct the speech signal. The time domain convolutional network model consists of two fully connected layers and N residual modules, and the input feature is the amplitude spectrum feature of noisy speech; The input features are normalized and then reshaped into a tensor of [number of sentences, sentence length, 1, 257] for model training or inference, where the fourth dimension of the tensor 257 corresponds to the amplitude spectrum features corresponding to each frame of speech data extracted. Subsequently, the transformed input features are obtained through a 256-dimensional fully connected layer. In order to better represent the input features, nonlinear rectifier units are used for nonlinear activation. Next, considering the temporal dependency between speech signals, the information between different frames is captured by stacking residual modules with hole convolutions, and the context information in the past frames is used to estimate the prior signal-to-noise ratio of the current frame. In terms of the design of the residual module, each residual module consists of three layers of convolutional layers. In order to reduce the number of parameters and the amount of computation, a bottleneck structure design is adopted. The output channel dimensions of the three convolutional layers are 128, 128, and 256, respectively, and the corresponding tensor sizes are [number of sentences, sentence length, 1, 128], [number of sentences, sentence length, 1, 128], and [number of sentences, sentence length, 1, 128], respectively. length, 1, 256], in order to further reduce the number of parameters and computational complexity of the model, the first and third convolutional layers are grouped from left to right, and the number of groups is set to 8. After the grouping operation, the channel dimension of each group is 16, and then 1×1 convolution is used to concatenate the channel information in each group. In order to ensure the information interaction between groups, the output of the first layer of grouped convolution is channel mixed. In each group, the channels corresponding to the same index are arranged from left to right, and then a convolution kernel of 1 is performed on each channel. ×3 dilated convolution operation, and the dilation rate is configured in a cyclic manner of 1, 2, 4, 8, and 16 for different residual modules. The last convolution layer performs the same grouped convolution operation as the first convolution layer, and then adds the input and output features of the residual module through skip connections. At the end of the entire time domain convolutional network model, a 257-dimensional fully connected layer is used to estimate the prior signal-to-noise ratio. Of course, in order to ensure that its output range is between [0,1], the sigmoid function is used as the activation function of the fully connected layer.

2. The lightweight time-domain convolutional network speech enhancement method according to claim 1, characterized in that: The reconstruction of speech signals includes inverse Fourier transform and overlap-add.

3. The lightweight time-domain convolutional network speech enhancement method according to claim 1, characterized in that: The construction process of the time domain convolutional network model is as follows: 1), according to the definition of the priori signal-to-noise ratio, its value range is any number greater than zero, as shown in formula (1): Among them, |X clean (n,k)| and |N noise (n,k)| represents the amplitude spectrum of pure speech and noise signal respectively, n and k are frame index and frequency index respectively. In order to reduce the learning difficulty of the time domain convolutional network model, its value range is compressed. Through statistical research on the prior signal-to-noise ratio, it is found that its logarithmic transformation form, as shown in formula (2), obeys Gaussian distribution. Therefore, a cumulative distribution function is used to compress it, as shown in formula (3): x dB (n,k)=10·log 10 (ξ(n,k)) (2) Among them, ξ dB (n, k) is the prior signal-to-noise ratio after logarithmic compression, and It is the prior signal-to-noise ratio after compression by the cumulative distribution function, and its value range is [0,1]. erf(·) represents the cumulative distribution function, μ k and σ k are obtained through statistical analysis at different frequencies. dB The mean and variance corresponding to (n,k); 2), considering the temporal dependency between speech signals, a lightweight time-domain convolutional network model is designed using dilated convolution to learn the amplitude spectrum of noisy speech |X noisy (n,k)| and the prior signal-to-noise ratio The mapping relationship between them.

4. The lightweight time-domain convolutional network speech enhancement method according to claim 3, characterized in that: The loss function of the time domain convolutional network model training is calculated using cross entropy, as shown in equation (4), where y i and They are the ideal priori signal-to-noise ratio and the priori signal-to-noise ratio results predicted by the neural network output layer:

5. The lightweight time-domain convolutional network speech enhancement method according to claim 4, characterized in that: The gain function of the noise reduction filter is estimated by using the gain function expression derived from the Bayesian theory and the minimum mean square error criterion, as shown in the following formula (5): Among them, G(n,k) is the estimated gain function, ξ(n,k) is the estimated value of the prior signal-to-noise ratio, which is also the lower limit of the integral. However, since the prior signal-to-noise ratio output by the temporal convolutional network is a compressed value, it is necessary to perform an inverse transformation on it to obtain the prior signal-to-noise ratio required in equation (5). The inverse transformation process is shown in equations (6) and (7): Substituting the inverse transformed priori signal-to-noise ratio into the gain function expression in equation (5), we can obtain the gain value of the amplitude spectrum of each frame of noisy speech at each frequency point. By performing a dot multiplication with each frequency point, we can obtain the enhanced speech amplitude value. Finally, the phase spectrum of the noisy speech can be used to synthesize the final enhanced speech signal.

6. A lightweight time-domain convolutional network speech enhancement system, characterized by: The method comprises a readable storage medium in which execution instructions are stored. When the execution instructions are executed by a processor, the execution instructions are used to implement the method according to any one of claims 1 to 5.

Images