Reference-based speech enhancement methods
By introducing a reference-based method in the speech enhancement algorithm, using MFCC group matching and deep feature fusion, the problems of poor voice enhancement effect and information loss in the prior art are solved, and a more efficient voice enhancement effect is achieved.
Patent Information
- Application Number
- CN202210042390.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-14
- Publication Date
- 2025-04-18
- Estimated Expiration
- 2042-01-14
AI Technical Summary
The existing voice enhancement algorithm still has some information loss when restoring enhanced speech, and it is difficult to effectively utilize external references or prior information of the voice itself to improve the enhancement effect.
Using a reference-based speech enhancement algorithm, noise-free speech and reference speech are converted to the frequency domain through short-time Fourier transform, inter-frame correspondence is obtained using MFCC group matching, reference features are recombined, and depth features are extracted and fused through encoder and decoder. Finally, the noise-free speech spectrum is filtered using a multiplicative complex mask to obtain enhanced speech.
By leveraging the local correlation between noisy speech and reference speech, more effective speech enhancement is achieved, reducing information loss and improving the quality of enhanced speech.
Smart Images
Figure CN114360571B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of speech signal processing and relates to a speech enhancement algorithm. Background Art
[0002] Speech signals collected in real environments usually contain background noise. These noises come in various forms, and their presence seriously reduces the auditory perception quality and intelligibility of speech. The purpose of speech enhancement is to extract useful speech signals from signals that are interfered with or even drowned by various noises, while suppressing and reducing the interference of noise.
[0003] Existing single-channel speech enhancement methods can be divided into two categories: one is traditional methods, such as spectral subtraction, Wiener filtering, methods based on hidden Markov models, methods based on non-negative matrices, etc. The other is methods based on deep learning.
[0004] Single-channel speech enhancement methods based on deep learning are mainly divided into time domain and frequency domain methods. Time domain methods usually use deep neural networks to learn the mapping relationship between the one-dimensional waveforms of noisy speech and clean speech. Since one-dimensional speech signals lack obvious geometric structures, it is difficult to extract useful information from them. Frequency domain methods process the two-dimensional spectrum of noisy speech signals in two ways. One is to directly reconstruct the clean speech spectrum through a deep neural network, and the other is to predict a time-frequency mask to filter the noisy spectrum to generate an enhanced spectrum. Although the emergence of the above methods has greatly improved the effect of single-channel speech enhancement methods based on deep learning, the restored enhanced speech still has some information lost. In order to further improve the effect of speech enhancement algorithms, many methods that use external reference or inherent prior information of speech to help reconstruct enhanced speech have been proposed, such as the mouth movement prior and expression prior in the external reference speaker video, the speaker identity prior, emotion prior, and phoneme pronunciation feature prior inherent in the speech itself or in the external clean reference speech. Among them, since the external reference speech and the target speech have the same speaker (different speaking content) and their frame-level phoneme information is partially shared, better results can be achieved by introducing external reference speech to assist in the enhancement of noisy speech (reference-based speech enhancement). Summary of the invention
[0005] In order to further improve the effect of existing speech enhancement algorithms, the present invention aims to provide a set of reference-based speech enhancement algorithms. The speech involved in the present invention is all single-channel speech. The technical solution is as follows
[0006] A reference-based speech enhancement method comprises the following steps:
[0007] 1) Analyze noisy speech in real environment and establish additive noise model:
[0008] x=y+n
[0009] Among them, y is clean speech, n is environmental noise, and x is noisy speech;
[0010] 2) Define a reference-based speech enhancement model:
[0011]
[0012] Where g is a reference-based speech enhancement model; r is the reference speech corresponding to the noisy speech x, that is, the clean speech with the same speaker as the noisy speech x but with different possible speech content; is the enhanced speech output by the model, that is, an estimate of the clean speech y;
[0013] 3) The noisy speech x and the reference speech r are transformed from the time domain to the frequency domain through short-time Fourier transform to obtain the noisy speech spectrum With reference speech spectrum Where T x and T r are the number of frames of X and R respectively, and F is the number of points in the frequency domain;
[0014] 4) MFCC group matching obtains the correspondence between frames, the method is as follows:
[0015] MFCCs are extracted from the noisy speech spectrum X and the reference speech spectrum R respectively, and the MFCCs of three consecutive frames are combined to form an MFCC group of the noisy speech spectrum X and an MFCC group of the reference speech spectrum R; for a certain MFCC group of noisy speech, the cosine similarity between the MFCC groups of all reference speech and the MFCC groups of the reference speech is calculated, and the middle frame of the five MFCC groups with the highest cosine similarity is selected to form an inter-frame correspondence with the middle group of the MFCC group of the noisy speech Repeating this process can obtain an inter-frame correspondence for each frame of noisy speech;
[0016] 5) The encoder extracts deep features as follows
[0017] The encoder consists of three cascaded convolutional layers, each of which contains three parts: complex convolution, batch normalization, and LeakyReLu activation. The complex convolution downsamples the input in the frequency domain by controlling the step size. The same encoder is used to process X and R separately to obtain three pairs of noisy features of different depths. and reference features
[0018] 6) Reference speech feature reconstruction, the method is as follows:
[0019] For each Based on the corresponding inter-frame correspondence right Reorganized to form 5 Recombined reference features of the same dimension
[0020] 7) Two-stage feature fusion, the method is as follows:
[0021] In the first stage, the softmax function is used to and The inter-frame cosine similarity of the amplitude value is normalized to generate the weight coefficient Then based on the weight coefficient For 5 Perform convolution fusion to obtain
[0022] In the second stage, channel attention is used to and Fusion, get
[0023] 8) Decoding of deep features:
[0024] The decoder consists of 4 residual blocks and 3 cascaded transposed convolutional layers. Each residual block consists of complex convolution, ReLu activation, complex convolution, and ReLu activation. The first two transposed convolutional layers contain three parts: complex transposed convolution, batch normalization, and LeakyReLu activation. The last transposed convolutional layer only has complex transposed convolution operations. Each transposed convolution operation in the decoder can upsample the input in the frequency domain by controlling the step size. The decoder first converts Send it to 4 residual blocks to get Then, three transposed convolutional layers are used to upsample it step by step, and each time before entering the transposed convolutional layer, it is combined with the corresponding Merge together:
[0025]
[0026] Among them, decode i Refers to the i-th transposed convolutional layer;
[0027] Finally, the tanh activation function is used to activate the decoder output The amplitude spectrum is processed to obtain a multiplicative complex mask M:
[0028]
[0029] 9) Use the multiplicative complex mask M to filter the noisy speech spectrum X to obtain the enhanced speech spectrum
[0030] 10) Through the short-time Fourier transform inverse transform Convert from frequency domain to time domain to obtain the final enhanced speech
[0031] Technical features and effects of the present invention: For reference-based speech enhancement, the present invention uses the inter-frame correspondence obtained based on MFCC group matching to reconstruct reference features. The reconstructed reference features and noise features are fused together through a feature fusion module and passed through a decoder to finally obtain an enhanced speech spectrum. The present invention has the following features:
[0032] 1. The present invention utilizes the local correlation between noisy speech and reference speech to perform reference-based speech enhancement. The present invention first uses MFCC groups for matching, and then reconstructs reference features based on the inter-frame correspondence obtained by matching.
[0033] 2. Since the reference features and the noisy features have different similarities and their contributions to the final enhanced speech are different, the present invention uses a feature fusion module to adaptively fuse the reference and noisy features together.
[0034] 3. The present invention is the first to carry out reference-based single-channel speech enhancement. It is hoped that the present invention can inspire more research on reference-based speech processing methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flowchart of the algorithm.
[0036] Figure 2 It is a model structure diagram.
[0037] Figure 3 It is the structural diagram of the fusion module.
[0038] Figure 4 This is a comparison chart of the impact of reference speech of different lengths on the enhancement results. DETAILED DESCRIPTION
[0039] The present invention adopts the following technical solution:
[0040] 1) Analyze noisy speech in real environment and establish additive noise model:
[0041] x=y+n
[0042] Among them, y is clean speech, n is environmental noise, and x is noisy speech.
[0043] 2) Define the reference-based speech enhancement model and its input and output:
[0044]
[0045] Where g is a reference-based speech enhancement model; r is the reference speech corresponding to x, that is, a clean speech with the same speaker as x but different speech content; is the enhanced speech output by the model, that is, an estimate of the clean speech y.
[0046] 3) Design algorithm flow for additive noise model.
[0047] The reference-based speech enhancement algorithm provided by the present invention comprises the following 8 steps:
[0048] Step 1: Use short-time Fourier transform to convert the noisy speech x and reference speech r from the time domain to the frequency domain to obtain the noisy speech spectrum With reference speech spectrum Where T x and T r are the frame numbers of X and R respectively, and F is the number of points in the frequency domain.
[0049] Step 2: MFCC group matching to obtain the correspondence between frames.
[0050] In order to obtain the inter-frame correspondence, the present invention first extracts MFCC (Mel Frequency Cepstrum Coefficient) from the spectrum and combines the MFCCs of three consecutive frames (previous frame, current frame, and next frame) to form an MFCC group. For a certain MFCC group of noisy speech, by calculating the cosine similarity between the MFCC groups of all reference speech and its MFCC group, the middle frame of the five MFCC groups with the highest cosine similarity is selected and the middle frame of the noisy MFCC group is used to form an inter-frame correspondence. By repeating this process, an inter-frame correspondence can be obtained for each frame of noisy speech.
[0051] Step 3: The encoder extracts deep features.
[0052] The encoder consists of three cascaded convolutional layers. Each convolutional layer contains three parts: complex convolution, batch normalization, and LeakyReLu activation. The complex convolution downsamples the input in the frequency domain by controlling the step size. Using the same encoder to process X and R separately can obtain three pairs of features with different depths. and These are called noisy features and reference features. The encoding process is as follows:
[0053]
[0054]
[0055] Among them, encode i is the i-th convolutional layer.
[0056] Step 4: Reconstruction based on reference speech features.
[0057] For each Based on the corresponding inter-frame correspondence right Reorganized to form 5 Recombined reference features of the same dimension
[0058] Step 5: Feature fusion.
[0059] In order to and Combined together, the present invention designs a two-stage feature fusion module:
[0060] In the first stage, the softmax function is used to and The inter-frame cosine similarity of the amplitude value is normalized to generate the weight coefficient
[0061]
[0062]
[0063]
[0064] in, and They are and The tth frame of |·| is the amplitude of the complex number, α t yes The vector composed.
[0065] Then based on the weight coefficient For 5 Perform convolution fusion to obtain
[0066]
[0067]
[0068] Among them, complexConv is a complex convolution.
[0069] In the second stage, channel attention is used to and For fusion, channel attention has various forms. The present invention adopts the following forms of channel attention:
[0070]
[0071]
[0072]
[0073]
[0074]
[0075] Among them, [·] is the and Merge in the channel dimension, GlobalPooling is the global pooling layer.
[0076] Step 6: Deep feature decoding.
[0077] The decoder consists of 4 residual blocks and 3 cascaded transposed convolutional layers. Each residual block consists of complex convolution, ReLu activation, complex convolution, and ReLu activation. The first two transposed convolutional layers contain three parts: complex transposed convolution, batch normalization, and LeakyReLu activation. The last transposed convolutional layer only has complex transposed convolution operations. Each transposed convolution operation in the decoder can upsample the input in the frequency domain by controlling the step size. The decoder first converts Send it to 4 residual blocks to get Then, three transposed convolutional layers are used to upsample it step by step, and each time before entering the transposed convolutional layer, it is combined with the corresponding Merge together:
[0078]
[0079] Among them, decode i Refers to the i-th transposed convolutional layer.
[0080] Finally, the tanh activation function is used to activate the decoder output The amplitude spectrum is processed to obtain a multiplicative complex mask M:
[0081]
[0082] Step 7: Filter the noisy speech spectrum X using the multiplicative complex mask M to obtain the enhanced speech spectrum
[0083] Step 8: Inverse short-time Fourier transform Convert from frequency domain to time domain to obtain the final enhanced speech
[0084] 4) Dataset acquisition and preprocessing.
[0085] The present invention uses the Voice Bank+DEMAND open source dataset to train and test the model. The training set of the Voice Bank+DEMAND open source dataset contains multiple noisy-clean speech data pairs with different contents from 28 speakers, totaling 11,572 pairs. The test set contains multiple noisy-clean speech data pairs with different contents from 2 speakers, totaling 824 pairs. For each noisy speech, 15 seconds of the corresponding reference speech is randomly selected from other clean speeches of the speaker with different contents.
[0086] 5) Training.
[0087] In the present invention, the loss function used in the training model is the weighted sum of L1-Loss and perceptual loss, the optimizer uses the Adam optimizer, and the initial learning rate is set to 0.0001. The present invention uses the deep learning framework Pytorch platform to train the model, iterates 80 times on the entire data set, then reduces the learning rate to 0.00001, iterates 20 times, so that the loss converges, and obtains the final model.
[0088] 6) Testing.
[0089] The noisy speech in the test set and its corresponding reference speech are input into the model to obtain the corresponding enhanced speech.
[0090]
[0091] Table 1
[0092] Table 1 is a comparison table of the results of the algorithm proposed in this invention and other speech enhancement algorithms on the Voice Bank+DEMAND test set. The references of each method in the table are as follows:
[0093] [1]Pascual S,Bonafonte A,Serra J.SEGAN: Speech enhancement generative adversarial network[J].arXiv preprint arXiv:1703.09452,2017.
[0094] [2]Germain FG,Chen Q,Koltun V.Speech denoising with deep featurelosses[J].arXiv preprint arXiv:1806.10522,2018.
[0095] [3]Fu S W,Liao C F,Tsao Y,et al.Metricgan:Generative adversarialnetworks based black-box metric scores optimization for speech enhancement[C] / / International Conference on Machine Learning.PMLR,2019: 2031-2041.
[0096] [4]Yin D,Luo C,Xiong Z,et al.PHASEN:A phase-and-harmonics-awarespeech enhancement network[C] / / Proceedings of the AAAI Conference onArtificial Intelligence.2020,34(05):9458-9465.
[0097] [5]Kim J,El-Khamy M,Lee J.T-gsa:Transformer with gaussian-weightedself-attention for speech enhancement[C] / / ICASSP 2020-2020IEEE InternationalConference on Acoustics,Speech and Signal Processing (ICASSP).IEEE,2020:6649-6653.
[0098] [6]Defossez A,Synnaeve G,Adi Y.Real time speech enhancement in thewaveform domain[J].arXiv preprint arXiv:2006.12847,2020.
[0099] [7]Fu S W,Yu C,Hsieh T A,et al.MetricGAN+:An Improved Version ofMetricGAN for Speech Enhancement[J].arXiv preprint arXiv:2104.03538,2021.
[0100] [8]Hsieh TA,Yu C,Fu SW,et al.Improving Perceptual Quality by Phone-Fortified Perceptual Loss using Wasserstein Distance for Speech Enhancement[J].arXiv preprint arXiv:2010.15174,2020.
[0101] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A reference-based speech enhancement method comprising the following steps: 1) Analyze noisy speech in real environment and establish additive noise model: x=y+n in, y is clean speech, n is environmental noise, and x is noisy speech; 2) Define a reference-based speech enhancement model: Where g is a reference-based speech enhancement model; r is the reference speech corresponding to the noisy speech x, that is, the clean speech with the same speaker but different speech content as the noisy speech x; is the enhanced speech output by the model, that is, an estimate of the clean speech y; 3) The noisy speech x and the reference speech r are transformed from the time domain to the frequency domain through short-time Fourier transform to obtain the noisy speech spectrum With reference speech spectrum Where T x and T r are the number of frames of X and R respectively, and F is the number of points in the frequency domain; 4) MFCC group matching obtains the correspondence between frames, the method is as follows: MFCCs are extracted from the noisy speech spectrum X and the reference speech spectrum R respectively, and the MFCCs of three consecutive frames are combined to form an MFCC group of the noisy speech spectrum X and an MFCC group of the reference speech spectrum R; for a certain MFCC group of noisy speech, the cosine similarity between the MFCC groups of all reference speech and the MFCC groups of the reference speech is calculated, and the middle frame of the five MFCC groups with the highest cosine similarity is selected to form an inter-frame correspondence with the middle group of the MFCC group of the noisy speech Repeating this process can obtain an inter-frame correspondence for each frame of noisy speech; 5) The encoder extracts deep features as follows The encoder consists of three cascaded convolutional layers, each of which contains three parts: complex convolution, batch normalization, and LeakyReLu activation. The complex convolution downsamples the input in the frequency domain by controlling the step size. The same encoder is used to process X and R separately to obtain three pairs of noisy features of different depths. and reference features 6) Reference speech feature reconstruction, the method is as follows: For each Based on the corresponding inter-frame correspondence right Reorganized to form 5 Recombined reference features of the same dimension 7) Two-stage feature fusion, the method is as follows: In the first stage, the softmax function is used to and The inter-frame cosine similarity of the amplitude value is normalized to generate the weight coefficient Then based on the weight coefficient For 5 Perform convolution fusion to obtain In the second stage, channel attention is used to and Fusion, get 8) Decoding of deep features: The decoder consists of 4 residual blocks and 3 cascaded transposed convolutional layers. Each residual block consists of complex convolution, ReLu activation, complex convolution, and ReLu activation. The first two transposed convolutional layers contain three parts: complex transposed convolution, batch normalization, and LeakyReLu activation. The last transposed convolutional layer only has complex transposed convolution operations. Each transposed convolution operation in the decoder can upsample the input in the frequency domain by controlling the step size. The decoder first Send it to 4 residual blocks to get Then, three transposed convolutional layers are used to upsample it step by step, and each time before entering the transposed convolutional layer, it is combined with the corresponding Merge together: Among them, decode i Refers to the i-th transposed convolutional layer; Finally, the tanh activation function is used to activate the decoder output The amplitude spectrum is processed to obtain a multiplicative complex mask M: 9) Use the multiplicative complex mask M to filter the noisy speech spectrum X to obtain the enhanced speech spectrum 10) Through the short-time Fourier transform inverse transform Convert from frequency domain to time domain to obtain the final enhanced speech