A method for single-channel speech enhancement
By embedding VGG19 module and FED-former model in U-Net network, combined with Fourier enhancement and histogram equalization processing, the problem of insufficient context information combination and long-term sequence prediction capabilities of single-channel speech enhancement model is solved, which significantly improves the problem of insufficient context information combination and long-term sequence prediction capabilities of single-channel speech enhancement model, The speech quality and intelligibility are achieved, especially in non-stationary noise environments.
Patent Information
- Application Number
- CN202210963653.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-08-11
AI Technical Summary
The existing single-channel speech enhancement model cannot effectively combine context information to extract deeper and richer feature information, and lacks ability to capture long-term sequence prediction and time series global nature, especially in non-stationary noise environments.
The VGG19 module is used as the encoding layer of the U-Net network, and the FED-former model is embedded in the decoder part. The spectrum diagram is equalized through Fourier enhancement module and histogram, combining the multi-head attention mechanism and adaptive features fusion. A feature extraction model is constructed to improve frequency domain mask output capability and reduce long-term series prediction errors.
The performance of the speech enhancement model in non-stationary noise environment is improved, the speech distortion is significantly reduced, and the speech quality and intelligibility are improved, especially the speech enhancement effect under different signal-to-noise ratio conditions.
Smart Images

Figure CN115273886B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of speech enhancement, and in particular relates to a single-channel speech enhancement method. Background Art
[0002] Traditional single-channel speech enhancement technology has been developed to a very mature stage, such as the widely used spectral subtraction method, Wiener filtering method, and optimal improved logarithmic amplitude spectrum estimation. However, due to the limitations of the traditional algorithm itself, it cannot handle non-stationary noise and low signal-to-noise ratio noise well. In recent years, speech enhancement technology based on deep neural networks has developed rapidly. Since there is no need to make assumptions about the data, but the intrinsic connection between them is obtained after processing large-scale data, it can estimate speech and noise and has made great progress in a stable noise environment. At present, many network models have been applied, such as deep neural networks, recurrent neural networks, convolutional neural networks, U-net neural networks, etc.
[0003] However, these models do not perfectly meet our requirements. The large number of parameters in deep neural network training makes it difficult to train, and it is impossible to use context information. Recurrent neural networks take a long time to train, the network size is large, and it is difficult to implement parallel processing. Convolutional neural networks predict the current clean speech signal by inputting the noisy speech signal of the previous frames, making good use of time correlation. Compared with recurrent neural networks, they have a smaller network size and shorter training time, and also generate fewer parameters than equivalent deep neural networks. However, convolutional neural networks have limited receptive fields and weak context modeling capabilities. On the other hand, the inherent feature extraction characteristics of convolutional neural networks encourage researchers to implement end-to-end speech enhancement networks in the time domain and waveform, but the speech effect of a single convolutional neural network structure is not ideal. On this basis, some scholars proposed a new structure based on U-Net, VGG19-RAFFUNet-HESE. This structure is based on the original U-Net structure and embeds the deep fully convolutional network VGG19 in the encoder part of the U-Net encoder and decoder network architecture; secondly, adaptive features are added between jump connections to fuse effective feature information, and finally a residual network is added to the decoder part to increase the network depth. This structure performs speech enhancement in the frequency domain. Experimental data show that in the case of unknown noise, the algorithm can reduce the distortion of the enhanced speech and obtain speech with better quality and intelligibility, with obvious enhancement effect.
[0004] However, the only drawback is that the residual neural network used in the decoder part of U-Net in the VGG19-RAFFUNet-HESE structure can only deepen the network depth to prevent network training degradation, but cannot help us combine contextual information, extract deeper and richer feature information, improve the ability to predict long-term sequences, and capture the global properties of time series. Summary of the invention
[0005] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a single-channel speech enhancement method, which is intended to solve the problem that the currently used extraction model cannot well combine contextual information, extract deeper and richer feature information, and improve the ability to predict long-term sequences and capture the global properties of time series.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:
[0007] A method for single-channel speech enhancement comprises the following steps:
[0008] Step 1: Obtain speech data and form a training set for training the U-Net network model;
[0009] Step 2: Perform discrete Fourier transform on the speech data in the training set through the Fourier enhancement module to obtain a spectrogram, and then perform histogram equalization enhancement on the spectrogram to obtain an enhanced spectrogram;
[0010] Step 3: Use the VGG19 network as the encoding layer of the U-Net network model, and embed the FED-former model into the decoder part of the U-Net network model to build a model to extract features;
[0011] Step 4: Extract features from the constructed model and train it, and process the enhanced spectrogram to be processed based on the trained model to obtain a clean speech signal.
[0012] Preferably, the speech data is synthesized from noisy speech and corresponding clean speech.
[0013] Preferably, the synthetic signal-to-noise ratio of the speech data is -10dB, -5dB, 0dB, 5dB, and 10dB.
[0014] Preferably, the noisy speech is obtained from the Noisex92 noise library; the clean speech is obtained from a training set of the TIMIT library.
[0015] Preferably, step 2 comprises the following steps:
[0016] Step 2.1: Perform discrete Fourier transform on the speech data in the training set, select a constant number of Fourier components to represent the sequence, and obtain a spectrogram;
[0017] Step 2.2: The unevenly distributed pixels of the spectrogram are converted into evenly distributed pixels through a mapping function, and then the pixels are calculated to obtain an enhanced spectrogram.
[0018] Preferably, the calculation steps for calculating the pixel points described in step 2.2 are as follows:
[0019] A: Calculate the probability function of the spectrogram distribution P(r K ):
[0020] P(r K )=N K / N,k=0,1,2…L-1;
[0021] Where: N K is the total number of occurrences of the kth gray level; N is the total number of pixels in the grayscale image; L is the number of gray levels;
[0022] B: According to P(r K ) Calculate the cumulative distribution function S K , through S K The enhancement is completed by mapping the new pixels and obtaining an enhanced spectrogram.
[0023] Preferably, the feature extraction model includes a VGG19 module, a hybrid expert decomposition module and an enhanced Fourier transform module;
[0024] The VGG19 module includes an input layer, a convolutional layer, a pooling layer, and a fully connected layer;
[0025] The hybrid expert decomposition module includes a plurality of average filters of different sizes, through which a plurality of trend components and a plurality of weights related to data are extracted from deep features; the trend components and the weights are then combined into an initial trend; the specific formula is as follows:
[0026] X trend = Softmax(L(x))*(F(x));
[0027] Where: X trend represents the initial trend; F(x) represents the output of the average pooling filter, and Softmax(L(x)) is the weight;
[0028] The enhanced Fourier transform module is a part of the FED-former model; each encoding network layer of the FED-former model includes a multi-head attention mechanism module, a fully connected layer module and a normalization module.
[0029] Preferably, the process of feature extraction based on the VGG19 module is as follows:
[0030] a. The input layer inputs the spectrogram. The spectrogram assumes that the input size is 256×256, and the VGG19 structure uses 5 convolution blocks;
[0031] b. In the first convolutional layer, the spectrogram is convolved to calculate the main features. The convolutional layer uses a 3×3 convolution kernel with 64 kernels. The output formula is:
[0032] Y f =P(Mat*w+b)
[0033] where Y f represents output, P represents activation function, Mat represents grayscale image matrix, * represents calculation between convolution kernel and grayscale image matrix, w represents convolution kernel, and b represents offset;
[0034] Then connect the activation function to add a nonlinear unit, and connect the pooling layer after the convolution. The pooling layer sets the maximum pooling box size to 2×2. The pooling layer needs to further extract the features of the spectrum graph to obtain a compressed feature map size of 128×128; this size of feature map is used as the input of the second convolution layer. At this time, the number of convolution kernels is 128, and so on for each subsequent layer.
[0035] c. Repeat step b 4 times to obtain the final compressed feature map;
[0036] d. The fully connected layer maps the compressed feature map obtained in step c to a feature vector of constant length;
[0037] The final result is that the five convolutional blocks of VGG19 act as a powerful deep feature extractor, first passing the shallow features obtained by downsampling each layer through the jump connection part, and then passing the deep features obtained by downsampling five times to the decoder part. The adaptive feature fusion (AFF) added in the jump connection part fuses the shallow features obtained by downsampling each layer of VGG19 with the deep features obtained by upsampling and downsampling at the same time.
[0038] The beneficial effects of the present invention include:
[0039] The present invention uses the VGG19 module as the encoder part of the U-Net network model, and embeds the FED-former model into the encoder and decoder parts of the U-Net network model to construct a feature extraction model, thereby improving the decoder's output capability for frequency domain masks and also reducing the decoder's prediction error for long time series. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a schematic diagram of the overall structure of the present invention.
[0041] Figure 2 Schematic diagram of the Fourier enhancement structure of the present invention.
[0042] Figure 3 This is the PESQ schematic diagram. DETAILED DESCRIPTION
[0043] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0044] The following is combined with Figure 1 and attached Figure 2 The embodiments of the present invention are further described as follows:
[0045] See attached Figure 1 , a method for single-channel speech enhancement, comprising the following steps:
[0046] Step 1: Acquire speech data and form a training set for training the U-Net network model; the speech data is synthesized from noisy speech and corresponding clean speech; the clean speech is obtained from the training set of the TIMIT library, 90 speakers' speech is selected, and the noise data comes from 20 real noises in the Noisex92 noise library; 2500 training data with synthesized signal-to-noise ratios of -10dB, -5dB, 0dB, 5dB, and 10dB.
[0047] Step 2: Perform discrete Fourier transform on the speech data in the training set through the Fourier enhancement module to obtain a spectrogram, and then perform histogram equalization enhancement on the spectrogram to obtain an enhanced spectrogram;
[0048] The step 2 comprises the following steps:
[0049] See attached Figure 2 ,Step 2.1: Perform discrete Fourier transform on the speech data in the training set, select a constant number of Fourier components to represent the sequence, and obtain a spectrogram;
[0050] Since Fourier analysis is a common tool for delving into the frequency domain, it is very important to know how to use Fourier analysis to appropriately represent the information in a time series. Since many high-frequency variations in time series are caused by noise input, simply retaining all frequency components may result in a poor representation. On the other hand, retaining only low-frequency components may also not be suitable for sequence prediction, because some trend changes in the time series represent important events. Instead, using a small number of selected Fourier components to keep the time series compact will lead to efficient computation of the Transformer model, which is crucial for long sequences. According to theoretical analysis, a randomly selected subset of frequency components, including low components and high components, can represent the time series well. So we choose to randomly select a constant number of Fourier components to represent the sequence, including low frequencies and high frequencies. Next, we verify why random selection is correct. Suppose there are m time series, denoted as X 1 (t), …… X m (t). First, apply the Fourier transform to each time series and reshape each X i (t) into a single-row matrix a i =(a i ,1, …… a i ,d) T ∈R d . By putting all the Fourier transform vectors into a matrix, we can get an A=(a 1 ,a 2 ……,a m ) T ∈R m×d , where each row corresponds to a different time series and each column corresponds to a different Fourier component. Although using all Fourier components can best preserve the historical information in the time series, it may lead to the omission of historical data, resulting in poor performance of the prediction signal. Therefore, it is necessary to carefully select a subset of Fourier components, which on the one hand should be small enough to avoid overfitting problems, and on the other hand should be able to preserve a large amount of historical information. So the invention proposes randomly selecting s components from d Fourier components, where s < d. More specifically, use i 1 <i 2 <…… i s to represent these randomly selected components, construct a matrix S∈{0,1} s×d , set S i,k = 0 if i = i k and S i,k = 0, and then the representation of the multivariate time series becomes A ’ = AS T ∈R m×sFinally, it is proved that even if the Fourier components are randomly selected, under the condition of neutralization, A ’ It can retain the information in A to the greatest extent.
[0051] Step 2.2: The unevenly distributed pixels of the spectrogram are converted into evenly distributed pixels through a mapping function, and the pixels are calculated to obtain an enhanced spectrogram;
[0052] The calculation steps for the pixel points described in step 2.2 are as follows:
[0053] A: Calculate the spectrogram distribution probability function P(rK):
[0054] P(r K )=N K / N,k=0,1,2…L-1;
[0055] Where: N K is the total number of occurrences of the kth gray level; N is the total number of pixels in the grayscale image; L is the number of gray levels;
[0056] B: According to P(r K ) Calculate the cumulative distribution function S K , S K Normalized to [0,255], each spectrogram is a 256×256 image, and an enhanced spectrogram is obtained.
[0057] Step 3: Use the VGG19 network as the encoding layer of the U-Net network model, and embed the FED-former model into the decoder part of the U-Net network model to build a model to extract features;
[0058] The feature extraction model includes a VGG19 module, a hybrid expert decomposition module and an enhanced Fourier transform module;
[0059] The VGG19 module includes an input layer, a convolutional layer, a pooling layer, and a fully connected layer;
[0060] The hybrid expert decomposition module includes multiple average filters of different sizes, and extracts multiple trend components and multiple data-related weights from deep features through the average filters; then the trend components and the weights are combined into an initial trend; the specific formula is as follows:
[0061] X trend = Softmax(L(x))*(F(x));
[0062] Where: X trend represents the initial trend; F(x) represents the output of the average pooling filter, and Softmax(L(x)) is the weight;
[0063] The enhanced Fourier transform module is the FEDformer model; each encoding network layer of the FEDformer model includes a multi-head attention mechanism module, a fully connected layer module and a normalization module; the feature extraction model
[0064] The convolution layer of the VGG19 module uses a (3×3) convolution kernel, and the maximum pooling size in the pooling layer is (2×2);
[0065] The process of feature extraction based on the VGG19 module is as follows:
[0066] a. The input layer inputs the spectrogram. The spectrogram assumes that the input size is 256×256, and the VGG19 structure uses 5 convolution blocks;
[0067] b. In the first convolutional layer, the spectrogram is convolved to calculate the main features. The convolutional layer uses a 3×3 convolution kernel with 64 kernels. The output formula is:
[0068] Y f =P(Mat*w+b);
[0069] where Y f represents output, P represents activation function, Mat represents grayscale image matrix, * represents calculation between convolution kernel and grayscale image matrix, w represents convolution kernel, and b represents offset;
[0070] Then connect the activation function to add a nonlinear unit, and connect the pooling layer after the convolution. The pooling layer sets the maximum pooling box size to 2×2. The pooling layer needs to further extract the features of the spectrum graph to obtain a compressed feature map size of 128×128; this size of feature map is used as the input of the second convolution layer. At this time, the number of convolution kernels is 128, and so on for each subsequent layer.
[0071] c. Repeat step b 4 times to obtain the final compressed feature map;
[0072] d. The fully connected layer maps the compressed feature map obtained in step c to a feature vector of constant length;
[0073] The final result is that the five convolutional blocks of VGG19 act as a powerful deep feature extractor, first passing the shallow features obtained by downsampling each layer through the jump connection part, and then passing the deep features obtained by downsampling five times to the decoder part. The adaptive feature fusion (AFF) added in the jump connection part fuses the shallow features obtained by downsampling each layer of VGG19 with the deep features obtained by upsampling and downsampling at the same time.
[0074] Step 4: Extract features from the constructed model and train it, and process the enhanced spectrogram to be processed based on the trained model to obtain a clean speech signal.
[0075] The beneficial effects of the present invention are further described below through experimental analysis:
[0076] The evaluation indicators used are speech perceptual quality evaluation (PESQ) and short-term objective intelligibility of speech (STOI). Here we compare with the original model (VGG19-RAFFUNET), another fused U-Net model (TU-NET), and other different enhanced models.
[0077] (1) PESQ: In February 2001, ITU-T proposed the latest P.862 standard "Objective evaluation method for end-to-end speech quality and speech codec quality in narrowband telephone networks". The proposal adopts the speech quality evaluation PESQ (PESQ, Perceptual Evaltation of Speech Quality) algorithm. Compared with other popular algorithms, the PESQ algorithm not only takes into account the end-to-end delay and can evaluate different types of networks, but also adopts improved auditory model and cognitive model technology, and has good robustness to communication delay, environmental noise, etc. Its speech database consists of nine languages, collected from different real and simulated networks. It is the one with the highest correlation with subjective evaluation among the objective speech quality evaluation algorithms published by ITU so far. The range is between [-0.5, 4.5]. The higher the score, the better the speech quality.
[0078] See attached Figure 3 As shown:
[0079] 1. Preprocessing includes the synchronization of input and output voice signals, level regularization, framing and other processing steps.
[0080] 2. The purpose of speech signal analysis is to extract the required information and obtain feature representation parameters.
[0081] 3. The objective distortion amount is the total difference between the characteristic parameters of the original speech and the output speech, that is, the degree of distortion of the output speech to the original speech.
[0082] 4. In order to be consistent with the subjective evaluation level, the distortion amount obtained by objective evaluation is usually mapped to the scale representation of subjective evaluation, such as the 5-level representation of MOS. The mapping module completes this function. The mapping function definition is shown in Formula 3:
[0083]
[0084] x is the original model output.
[0085] (2) STOI: Short-Time Objective Intelligibilty is an important indicator for measuring speech intelligibility. For a word in a speech signal, there are only two situations: it can be understood or it cannot be understood. From this perspective, intelligibility can be considered to be binary, so the value range of STOI is quantified between [0,1], representing the percentage of words that are correctly understood. The higher the value, the better the speech enhancement effect.
[0086] The first two rows in the table show the changes in signal-to-noise ratio and noisy speech, and the following rows show the comparison results of PESQ and STOI of different algorithms.
[0087] Table 1 PESQ comparison results of different algorithms
[0088]
[0089] Table 2 STOI comparison results of different algorithms
[0090]
[0091] From the experimental results, it can be seen that under different signal-to-noise ratio conditions, the VAFED-Net model constructed by the present invention has better speech quality and speech enhancement effects than other models.
[0092] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.
Claims
1. A method for single-channel speech enhancement, It is characterized in that The following steps are involved: Step 1: Obtain speech data and form a training set for training the U-Net network model; Step 2: Perform discrete Fourier transform on the speech data in the training set through the Fourier enhancement module to obtain a spectrogram, and then perform histogram equalization enhancement on the spectrogram to obtain an enhanced spectrogram; Step 2.1: Perform discrete Fourier transform on the speech data in the training set, select a constant number of Fourier components to represent the sequence, and obtain a spectrogram; Step 2.2: The unevenly distributed pixels of the spectrogram are converted into evenly distributed pixels through a mapping function, and then the pixels are calculated to obtain an enhanced spectrogram; Step 3: Use the VGG19 network as the encoding layer of the U-Net network model, and embed the FED-former model into the decoder part of the U-Net network model to construct a feature extraction model; the feature extraction model includes a VGG19 module, a hybrid expert decomposition module, and an enhanced Fourier transform module; The VGG19 module includes an input layer, a convolutional layer, a pooling layer, and a fully connected layer; The hybrid expert decomposition module includes a plurality of average filters of different sizes, and extracts a plurality of trend components and a plurality of weights related to the data from the deep features through the average filters; and then merges the trend components and the weights into an initial trend; The specific formula is as follows: X trend = Softmax(L(x))*(F(x)); Where: X trend represents the initial trend; F(x) represents the output of the average pooling filter, and Softmax(L(x)) is the weight; The enhanced Fourier transform module is a part of the FED-former model; each encoding network layer of the FED-former model includes a multi-head attention mechanism module, a fully connected layer module and a normalization module; Step 4: Train the constructed feature extraction model, and process the enhanced spectrogram to be processed based on the trained model to obtain a clean speech signal.
2. A method for single-channel speech enhancement according to claim 1, It is characterized in that The speech data is synthesized from noisy speech and corresponding clean speech.
3. A method for single-channel speech enhancement according to claim 2, It is characterized in that The synthesized signal-to-noise ratio of the speech data is -10dB, -5dB, 0dB, 5dB, and 10dB.
4. The method for single-channel speech enhancement according to claim 2, It is characterized in that The noisy speech is obtained from the Noisex92 noise library; the clean speech is obtained from the training set of the TIMIT library.
5. The method for single-channel speech enhancement according to claim 1, It is characterized in that The calculation steps for the pixel points described in step 2.2 are as follows: A: Calculate the probability function of the spectrogram distribution P(r K ): P(r K )=N K / N,k=0,1,2…L-1; Where: N K is the total number of occurrences of the kth gray level; N is the total number of pixels in the grayscale image; L is the number of gray levels; B: According to P(r K ) Calculate the cumulative distribution function S K , through S K The enhancement is completed by mapping the new pixels and obtaining an enhanced spectrogram.
6. A single-channel speech enhancement method according to claim 1, It is characterized in that The process of feature extraction based on the VGG19 module is as follows: a. The input layer obtains the enhanced spectrogram; b. Perform convolution operation on the enhanced spectrogram through the convolution layer, fill the feature map output by the convolution layer, keep the size of the feature map unchanged, and then connect the ReLU function to introduce the nonlinear unit; connect the size layer after the convolution layer to perform maximum pooling on the padded feature map to obtain a compressed feature map; c. Repeat step b 4 times to obtain the final compressed feature map; d. The fully connected layer maps the compressed feature map obtained in step c to a feature vector of unchanged length.
