A speech enhancement method, apparatus, device and medium

Abstract

The application discloses a speech enhancement method, device and equipment and a medium, and relates to the technical field of speech processing, and can improve speech quality and intelligibility.

Description

TECHNICAL FIELD

[0001] The present application relates to a speech enhancement method, device, equipment and medium, and belongs to the technical field of speech signal processing. BACKGROUND

[0002] Speech enhancement is a processing technology for noisy speech to improve speech quality and intelligibility, and the main goal is to eliminate the interference of background noise on speech.

[0003] Traditional speech enhancement techniques include Wiener filter, spectral subtraction and minimum mean square error estimation, etc. These methods are based on the assumption that the background noise is stationary in the denoising principle. Therefore, the effect of traditional speech enhancement techniques in processing noisy speech under non-stationary noise conditions is often not good. Under real environmental noise conditions, such traditional speech enhancement techniques are difficult to meet the actual application requirements.

[0004] With the rapid development of deep learning technology, various neural network structures are used in speech enhancement methods under non-stationary noise conditions, and good results have been achieved, such as dilated convolution network, attention mechanism network and convolutional recurrent network, etc. However, these neural networks only simply predict speech in the training process, ignoring the influence of noise information on speech enhancement technology.

[0005] In order to further promote the orthogonal separation of speech and noise, the scale-invariant signal-to-noise ratio loss (SI-SNR Loss) is proposed to train the neural network, which has been widely used in advanced network structures and has achieved good results. However, when this loss is used to train the neural network, it does not limit the scale of the speech, so it may cause the amplitude of the enhanced speech to be too large to form clipping distortion, thereby causing the quality and intelligibility of the speech to decrease. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art, provide a speech enhancement method, device, equipment and medium, and solve the technical problems that the speech enhancement effect of the prior art is not good, and the quality and intelligibility of the speech cannot meet the actual application requirements.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a speech enhancement method, comprising:

[0009] obtaining the time-domain waveform of the speech to be enhanced;

[0010] input the to-be-enhanced voice into the trained double-branch complex convolution recurrent neural network to obtain time-domain waveforms of the voice branch and the noise branch;

[0011] output the time-domain waveforms of the voice branch as an enhancement result;

[0012] The training of the double-branch complex convolution recurrent neural network comprises:

[0013] obtain time-domain waveforms of a preset number of noisy voices, and time-domain waveforms of pure noise voices and clean voices corresponding to the noisy voices;

[0014] construct a double-branch complex convolution recurrent neural network, input the time-domain waveforms of the noisy voices into the constructed double-branch complex convolution recurrent neural network, and obtain time-domain waveforms of the voice branch and the noise branch;

[0015] calculate a first scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the pure noise voices corresponding to the noisy voices and the noise branch, calculate a second scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the clean voices corresponding to the noisy voices and the voice branch, and obtain a total loss by summing the first scale-sensitive signal-to-noise ratio loss and the second scale-sensitive signal-to-noise ratio loss;

[0016] optimize the double-branch complex convolution recurrent neural network by using a gradient descent algorithm according to the total loss until a preset training termination condition is reached.

[0017] Optionally, the double-branch complex convolution recurrent neural network comprises a differentiable Fourier transform module, a complex convolution module group of the voice branch and the noise branch, a first bridge module group, a complex long-short term memory module group of the voice branch and the noise branch, a second bridge module group, a complex deconvolution module group of the voice branch and the noise branch, and a differentiable inverse Fourier transform module.

[0018] The differentiable Fourier transform module is implemented by a convolution layer and is configured to convert the time-domain waveforms of the input voice into complex time-frequency spectrum features;

[0019] The complex convolution module group of the voice branch and the noise branch each comprises N layers of complex convolution modules connected in a head-to-tail manner, the first bridge module group comprises N layers of bridge modules, and each bridge module of the first bridge module group is configured to connect the corresponding layers of complex convolution modules of the voice branch and the noise branch; the complex convolution module groups of the voice branch and the noise branch are respectively configured to process the complex time-frequency spectrum features to obtain one-time complex features of the voice branch and the noise branch.

[0020] The complex long short-term memory module group of the speech branch and the noise branch each includes M layers of complex long short-term memory modules connected in a head-to-tail manner, and the complex long short-term memory module of the speech branch and the complex long short-term memory module of the noise branch are respectively used for processing the first complex feature of the speech branch and the first complex feature of the noise branch to obtain the second complex feature of the speech branch and the second complex feature of the noise branch.

[0021] The complex deconvolution module group of the speech branch and the complex deconvolution module group of the noise branch each includes N layers of complex deconvolution modules connected in a head-to-tail manner, the second bridge module group includes N layers of bridge modules, and each bridge module of the second bridge module group is used for connecting the corresponding layer of complex deconvolution modules of the speech branch and the noise branch; the kth layer of complex deconvolution modules of the speech branch and the noise branch is connected with the N-k+1th layer of complex convolution modules of the speech branch and the noise branch; and the complex deconvolution module group of the speech branch and the complex deconvolution module group of the noise branch are respectively used for processing the second complex feature of the speech branch and the second complex feature of the noise branch to obtain the third complex feature of the speech branch and the third complex feature of the noise branch.

[0022] The complex time-frequency spectrum feature is subjected to a complex multiplication operation with the third complex feature of the speech branch and the third complex feature of the noise branch to obtain a complex multiplication operation result of the speech branch and the noise branch.

[0023] The differentiable inverse Fourier transform module is implemented by a convolution layer and is used for converting the complex multiplication operation result of the speech branch and the noise branch into a time domain waveform of the input speech in the speech branch and the noise branch.

[0024] Optionally, the bridge module includes two complex convolution modules, and the expressions of the two complex convolution modules are Bridge s and Bridge n .

[0025] Bridge s = cConvModule s (X n )+X s

[0026] Bridge n = cConvModule n (X s )+X n

[0027] In the expressions, X s and X n are inputs of the bridge module in the speech branch and the noise branch; cConvModule s and cConvModule n are the inputs X n and X sBridge s , Bridge n Bridge

[0028] Optionally, the complex convolution module comprises a complex convolution layer, a complex batch normalization layer and a complex PReLU layer connected in sequence.

[0029] The expression S1 of the complex convolution layer is:

[0030]

[0031] In the formula, X and S1 are the input and output of the complex convolution layer, X r , X i are the real part and the imaginary part of the input X, W is the convolution kernel of the complex convolution layer, W r , W i are the real part and the imaginary part of the convolution kernel W, is a complex convolution operation, and * is a convolution operation.

[0032] The expression S2 of the complex batch normalization layer is:

[0033] S2 = cBN (S1) = BN (S 1r ) + iBN (S 1i )

[0034] In the formula, S2 is the output of the complex batch normalization layer, cBN is a complex batch normalization operation, BN is a batch normalization operation, S 1r , S 1i are the real part and the imaginary part of S1.

[0035] The expression S3 of the complex PReLU layer is:

[0036] S3 = cPReLU (S2) = PReLU (S 2r ) + iPReLU (S 2i )

[0037] In the formula, S3 is the output of the complex PReLU layer, cPReLU is a complex PReLU operation, PReLU is a PReLU operation, S 2r , S 2i are the real part and the imaginary part of S2.

[0038] Optionally, the expression S4 of the complex long short-term memory module is:

[0039] S4 = cLSTM (X ′ ) = (out rr -out ii ) + i (outri -out ir )

[0040] out rr = LSTM r (X r ′ )

[0041] out ir = LSTM r (X i ′ )

[0042] out ri = LSTM i (X r ′ )

[0043] out ii = LSTM i (X i ′ )

[0044] wherein X ′ , S4 are the input and output of the complex long short-term memory module, X r ′ , X i ′ are the real and imaginary parts of the input X ′ , cLSTM is the complex long short-term memory operation, LSTM r , LSTM i are the real and imaginary parts long short-term memory operation.

[0045] Optionally, the complex deconvolution module comprises a complex deconvolution layer, a complex batch normalization layer and a complex PReLU layer connected in sequence.

[0046] The expression S5 of the complex convolution layer is:

[0047]

[0048] wherein X'', S5 are the input and output of the complex deconvolution layer, X r '', X i '' are the real and imaginary parts of the input X'', W'' is the deconvolution kernel of the complex convolution layer, W r '', W i '' are the real and imaginary parts of the deconvolution kernel W'', cDeconv is the complex deconvolution operation, Deconv is the deconvolution operation.

[0049] The expression S6 of the complex batch normalization layer is: ​​

[0050] S6 = cBN(S5) = BN(S 5r )+iBN(S 5i )

[0051] where S6 is the output of the complex batch normalization layer, cBN is the complex batch normalization operation, BN is the batch normalization operation, S 5r , S 5i are the real and imaginary parts of S5, respectively;

[0052] The expression S7 of the complex PReLU layer is:

[0053] S7 = cPReLU(S6) = PReLU(S 6r )+iPReLU(S 6i )

[0054] where S7 is the output of the complex PReLU layer, cPReLU is the complex PReLU operation, PReLU is the PReLU operation, S 6r , S 6i are the real and imaginary parts of S6, respectively.

[0055] Optionally, the expression SA_SNRLoss of the total loss is:

[0056]

[0057]

[0058]

[0059]

[0060] where are the first and second scale-sensitive signal-to-noise ratio losses, s1, are the time-domain waveforms of the pure noise speech and the noise branch corresponding to the noisy speech, s2, are the time-domain waveforms of the clean speech and the speech branch corresponding to the noisy speech, |·| is an L1 norm calculation operator, and |·| 2 is an L2 norm calculation operator, is s m is a dot product operation, and m = 1, 2.

[0061] In a second aspect, the present application provides a speech enhancement device, which comprises:

[0062] a speech acquisition module, configured to acquire the time-domain waveform of the speech to be enhanced;

[0063] The voice enhancement module is configured to input the to-be-enhanced voice into the trained double-branch complex convolution recurrent neural network to obtain time-domain waveforms of a voice branch and a noise branch.

[0064] The voice output module is configured to output the time-domain waveforms of the voice branch as an enhancement result.

[0065] The training of the double-branch complex convolution recurrent neural network comprises:

[0066] The data acquisition module is configured to acquire time-domain waveforms of a preset number of noisy voices, and time-domain waveforms of pure noise voices and clean voices corresponding to the noisy voices.

[0067] The network training module is configured to construct a double-branch complex convolution recurrent neural network, input the time-domain waveforms of the noisy voices into the constructed double-branch complex convolution recurrent neural network, and obtain time-domain waveforms of a voice branch and a noise branch.

[0068] The loss calculation module is configured to calculate a first scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the pure noise voices corresponding to the noisy voices and the noise branch, calculate a second scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the clean voices corresponding to the noisy voices and the voice branch, and obtain a total loss by summing the first scale-sensitive signal-to-noise ratio loss and the second scale-sensitive signal-to-noise ratio loss.

[0069] The network optimization module is configured to optimize the double-branch complex convolution recurrent neural network by using a gradient descent algorithm according to the total loss until a preset training termination condition is reached.

[0070] In a third aspect, the present application provides an electronic device comprising a processor and a storage medium.

[0071] The storage medium is configured to store instructions.

[0072] The processor is configured to operate according to the instructions to perform the steps of the above method.

[0073] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0074] Compared with the prior art, the present application has the following beneficial effects:

[0075] This invention provides a speech enhancement method, apparatus, device, and medium. The method inputs a noisy speech time-domain waveform into a two-branch complex convolutional recurrent neural network (RNN) to obtain two time-domain waveforms from the outputs of the two branches. It calculates the scale-sensitive signal-to-noise ratio (SNR) loss of the two output time-domain waveforms compared to the clean speech time-domain waveform and the pure noisy speech time-domain waveform labels to optimize the RNN. The noisy speech is then input into the trained RNN to obtain enhanced speech from the speech branch. This invention effectively solves the problem of uncontrollable amplitude variation in enhanced speech caused by using scale-invariant SNR loss (SI-SNR Loss) or optimized scale-invariant SNR loss (OSI-SNR Loss), limiting the amplitude of enhanced speech within a normal range and significantly improving speech quality and intelligibility. The apparatus, device, and medium employing the above method can achieve the same technical effect. Attached Figure Description

[0076] Figure 1 This is a flowchart of a speech enhancement method provided in Embodiment 1 of the present invention;

[0077] Figure 2 This is a schematic diagram of the structure of the dual-branch complex convolutional recurrent neural network provided in Embodiment 1 of the present invention;

[0078] Figure 3 This is a schematic diagram of the bridging module provided in Embodiment 1 of the present invention;

[0079] Figure 4 This is a schematic diagram of the complex convolution module provided in Embodiment 1 of the present invention;

[0080] Figure 5 This is a schematic diagram of the complex long short-term memory module provided in Embodiment 1 of the present invention;

[0081] Figure 6 This is a schematic diagram of the complex deconvolution module provided in Embodiment 1 of the present invention. Detailed Implementation

[0082] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0083] Example 1:

[0084] like Figure 1 As shown, this embodiment of the invention provides a speech enhancement method, including an enhancement phase and a training phase:

[0085] The enhancement stage is:

[0086] S101, acquiring a time-domain waveform of a speech to be enhanced;

[0087] S102, inputting the speech to be enhanced into the trained double-branch complex convolution recurrent neural network to acquire time-domain waveforms of a speech branch and a noise branch;

[0088] S103, outputting the time-domain waveform of the speech branch as an enhancement result.

[0089] The training stage is:

[0090] S201, acquiring time-domain waveforms of a preset number of noisy speeches, and time-domain waveforms of pure noise speeches and clean speeches corresponding to the noisy speeches;

[0091] S202, constructing a double-branch complex convolution recurrent neural network, inputting the time-domain waveforms of the noisy speeches into the constructed double-branch complex convolution recurrent neural network to acquire time-domain waveforms of a speech branch and a noise branch;

[0092] S203, calculating a first scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the pure noise speeches corresponding to the noisy speeches and the noise branch, and calculating a second scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the clean speeches corresponding to the noisy speeches and the speech branch, summing the first scale-sensitive signal-to-noise ratio loss and the second scale-sensitive signal-to-noise ratio loss to acquire a total loss;

[0093] An expression of the total loss is SA_SNRLoss:

[0094]

[0095]

[0096]

[0097]

[0098] In the formula, is the first scale-sensitive signal-to-noise ratio loss and the second scale-sensitive signal-to-noise ratio loss, s1, is the time-domain waveform of the pure noise speech corresponding to the noisy speech and the noise branch, s2, is the time-domain waveform of the clean speech corresponding to the noisy speech and the speech branch, |·| is an L1 norm calculation symbol, and |·| 2 is an L2 norm calculation symbol, is s m is a dot product operation, and m=1, 2;

[0099] S204, optimizing the double-branch complex convolution recurrent neural network according to the total loss by using a gradient descent algorithm until a preset training termination condition is reached; the termination condition can be set as convergence of the total loss, the total loss being less than a set loss value, the number of training reaching more than a preset number, etc. In this embodiment, the learning rate of the gradient descent algorithm is set as 0.01, and is attenuated by 0.5 until 0.00025 when the loss increases, and the training is terminated.

[0100] As shown in Figure 2 The double-branch complex convolution recurrent neural network provided in this embodiment includes a differentiable Fourier transform module, a complex convolution module group of the speech branch and the noise branch, a first bridge module group, a complex long-short term memory module group of the speech branch and the noise branch, a second bridge module group, a complex deconvolution module group of the speech branch and the noise branch, and a differentiable inverse Fourier transform module.

[0101] (1) The differentiable Fourier transform module (Conv-STFT) is implemented by a convolution layer, and is used to convert the time-domain waveform of the input speech (i.e. the noisy speech) into a complex time-frequency spectrum feature.

[0102] (2) The complex convolution module group of the speech branch and the noise branch each includes N layers of complex convolution modules connected in a head-to-tail manner, the first bridge module group includes N layers of bridge modules, and each layer of the bridge modules of the first bridge module group is used to connect the corresponding layer of complex convolution modules of the speech branch and the noise branch; the complex convolution module groups of the speech branch and the noise branch are respectively used to process the complex time-frequency spectrum feature to obtain a first complex feature of the speech branch and the noise branch; in this embodiment, N=6.

[0103] (3) The complex long-short term memory module group of the speech branch and the noise branch each includes M layers of complex long-short term memory modules connected in a head-to-tail manner, and the complex long-short term memory modules of the speech branch and the noise branch are respectively used to process the first complex feature of the speech branch and the noise branch to obtain a second complex feature of the speech branch and the noise branch; in this embodiment, M=2.

[0104] (4) The complex deconvolution module group of the speech branch and the noise branch each includes N layers of complex deconvolution modules connected in a head-to-tail manner, the second bridge module group includes N layers of bridge modules, and each layer of the bridge modules of the second bridge module group is used to connect the corresponding layer of complex deconvolution modules of the speech branch and the noise branch; the kth layer of complex deconvolution modules of the speech branch and the noise branch is connected with the N-k+1th layer of complex convolution modules of the speech branch and the noise branch; the complex deconvolution module groups of the speech branch and the noise branch are respectively used to process the second complex feature of the speech branch and the noise branch to obtain a third complex feature of the speech branch and the noise branch.

[0105] (5) The complex time-frequency spectral features are multiplied by the cubic complex features of the speech branch and the noise branch respectively to obtain the results of the complex multiplication operation of the speech branch and the noise branch.

[0106] (6) The Differentiable Inverse Fourier Transform (Conv-iSTFT) module is implemented by a convolutional layer and is used to transform the complex multiplication results of the speech branch and the noise branch into the time-domain waveforms of the input speech in the speech branch and the noise branch (i.e. the estimated pure noise speech and clean speech).

[0107] like Figure 3 As shown, the bridging module provided in this embodiment includes two complex convolution modules, and the expression for the two complex convolution modules is Bridge. s Bridge n for:

[0108] Bridge s =cConvModule s (X n )+X s

[0109] Bridge n =cConvModule n (X s )+X n

[0110] In the formula, X s X n cConvModule serves as the input for the bridging module in the speech and noise branches. s cConvModule n To input X n X s Bridge, as the input complex convolution module s Bridge n This is the output of the bridging module in the speech and noise branches.

[0111] like Figure 4 As shown, the complex convolution module provided in this embodiment includes a complex convolution layer, a complex batch normalization layer, and a complex PReLU layer connected in sequence;

[0112] The expression S1 for the complex convolutional layer is:

[0113]

[0114] In the formula, X and S1 are the input and output of the complex convolutional layer, respectively. r X i Let X be the real and imaginary parts of the input X, and W be the convolution kernel of the complex convolutional layer.r W i Let W be the real and imaginary parts of the convolution kernel. is a complex convolution operation, and * is a convolution operation;

[0115] The expression S2 for the complex batch normalization layer is:

[0116] S2=cBN(S1=BN(S) 1r )+iBN(S 1i )

[0117] In the formula, S2 is the output of the complex batch normalization layer, cBN is the complex batch normalization operation, BN is the batch normalization operation, and S... 1r S 1i Let S1 be the real and imaginary parts;

[0118] The expression S3 for the complex PReLU layer is:

[0119] S3 = cPReLU(S2) = PReLU(S 2r )+iPReLU(S 2i )

[0120] In the formula, S3 is the output of the complex PReLU layer, cPReLU is the complex PReLU operation, PReLU is the PReLU operation, and S... 2r S 2i Let S be the real and imaginary parts of S2.

[0121] like Figure 5 As shown, the expression S4 of the complex long short-term memory module provided in this embodiment is:

[0122] S4 = cLSTM(X) ′ ) = (out rr -out ii )+i(out ri -out ir )

[0123] out rr =LSTM r (X r ′ )

[0124] out ir =LSTM r (X i ′ )

[0125] out ri =LSTM i (X r ′ )

[0126] out ii =LSTM i (X i ′ )

[0127] In the formula, X ′ S4 is the input and output of the complex long short-term memory module, X r ′ X i ′ For input X ′ The real and imaginary parts, cLSTM is a complex long short-time memory operation, LSTM r LSTM i This is for short-term and long-term memory operations for the real and imaginary parts.

[0128] like Figure 6 As shown, the complex deconvolution module provided in this embodiment includes a complex deconvolution layer, a complex batch normalization layer, and a complex PReLU layer connected in sequence.

[0129] The expression S5 for the complex convolutional layer is:

[0130]

[0131] In the formula, X″ and S5 are the input and output of the complex deconvolution layer, respectively. r "、X i " represents the real and imaginary parts of the input X", and W" represents the deconvolution kernel of the complex convolutional layer. r "、W i " represents the real and imaginary parts of the deconvolution kernel W". For complex deconvolution operations, This is a deconvolution operation;

[0132] The expression S6 for the complex batch normalization layer is:

[0133] S6=cBN(S5=BN(S) 5r )+iBN(S 5i )

[0134] In the formula, S6 is the output of the complex batch normalization layer, cBN is the complex batch normalization operation, BN is the batch normalization operation, and S... 5r S 5i For the real and imaginary parts of S5;

[0135] The expression S7 for the complex PReLU layer is:

[0136] S7 = cPReLU(S6) = PReLU(S 6r )+iPReLU(S 6i )

[0137] In the formula, S7 is the output of the complex PReLU layer, cPReLU is the complex PReLU operation, PReLU is the PReLU operation, S6 is the output of the complex PReLU layer, and S 6r , S 6i is the real part and the imaginary part of S6.

[0138] Taking the scale convention of the input speech as [B, C, t] as an example, B is the batch size, C is the number of channels, and t is the time index on the time domain. The working process of the double-branch complex convolution recurrent neural network is as follows:

[0139] (1) For the input speech [B, 1, t] on the time domain waveform, the Fourier transform module implemented by convolution is used, and the output scale convention is [B, 2, F, T], F is the index frequency, and T is the time index on the frequency domain;

[0140] (2) For the input data [B, 2, F, T] with a channel number of 2, a complex convolution module with a convolution kernel size of 3 and a step of [2, 1] is used, the channel number is increased to 16, and the input of the bridge module (the convolution kernel size is 1 and the step is 1) is introduced, to obtain an output with a scale of [B, 16, F / 2, T].

[0141] (3) For the input data [B, 16, F / 2, T] with a channel number of 16, a complex convolution module with a convolution kernel size of 3 and a step of [2, 1] is used, the channel number is increased to 32, and the input of the bridge module (the convolution kernel size is 1 and the step is 1) is introduced, to obtain an output with a scale of [B, 32, F / 4, T].

[0142] (4) For the input data [B, 32, F / 4, T] with a channel number of 32, a complex convolution module with a convolution kernel size of 3 and a step of [2, 1] is used, the channel number is increased to 64, and the input of the bridge module (the convolution kernel size is 1 and the step is 1) is introduced, to obtain an output with a scale of [B, 64, F / 8, T].

[0143] (5) For the input data [B, 64, F / 8, T] with a channel number of 64, a complex convolution module with a convolution kernel size of 3 and a step of [2, 1] is used, the channel number is increased to 128, and the input of the bridge module (the convolution kernel size is 1 and the step is 1) is introduced, to obtain an output with a scale of [B, 128, F / 16, T].

[0144] (6) For the input data [B, 128, F / 16, T] with a channel number of 128, a complex convolution module with a convolution kernel size of 3 and a step of [2, 1] is used, the channel number is increased to 256, and the input of the bridge module (the convolution kernel size is 1 and the step is 1) is introduced, to obtain an output with a scale of [B, 256, F / 32, T].

[0145] (7) For the input data [B, 256, F / 32, T] with 256 channels, the data is input into the complex long short-term memory module with two hidden layers of 1024 to obtain the output with scale invariance.

[0146] (8) For the input data with 256 channels, the data is concatenated with the output of the corresponding complex convolution module in the channel dimension, and then input into the complex deconvolution module with a convolution kernel size of 3 and a step of [2, 1], so that the channel number is reduced to 128, and the input of the bridge module (convolution kernel size of 1 and step of 1) is introduced to obtain the output with a scale of [B, 128, F / 16, T].

[0147] (9) For the input data [B, 128, F / 16, T] with 128 channels, the data is concatenated with the output of the corresponding complex convolution module in the channel dimension, and then input into the complex deconvolution module with a convolution kernel size of 3 and a step of [2, 1], so that the channel number is reduced to 64, and the input of the bridge module (convolution kernel size of 1 and step of 1) is introduced to obtain the output with a scale of [B, 64, F / 8, T].

[0148] (10) For the input data [B, 64, F / 8, T] with 64 channels, the data is concatenated with the output of the corresponding complex convolution module in the channel dimension, and then input into the complex deconvolution module with a convolution kernel size of 3 and a step of [2, 1], so that the channel number is reduced to 32, and the input of the bridge module (convolution kernel size of 1 and step of 1) is introduced to obtain the output with a scale of [B, 32, F / 4, T].

[0149] (11) For the input data [B, 32, F / 4, T] with 32 channels, the data is concatenated with the output of the corresponding complex convolution module in the channel dimension, and then input into the complex deconvolution module with a convolution kernel size of 3 and a step of [2, 1], so that the channel number is reduced to 16, and the input of the bridge module (convolution kernel size of 1 and step of 1) is introduced to obtain the output with a scale of [B, 16, F / 2, T].

[0150] (12) For the input data [B, 16, F / 2, T] with 16 channels, the data is concatenated with the output of the corresponding complex convolution module in the channel dimension, and then input into the complex deconvolution module with a convolution kernel size of 3 and a step of [2, 1], so that the channel number is reduced to 2, and the input of the bridge module (convolution kernel size of 1 and step of 1) is introduced to obtain the output with a scale of [B, 2, F, T].

[0151] (13) For the input data [B, 2, F, T] in the complex time-frequency domain, the data is input into the inverse Fourier transform module realized by convolution to obtain the output with a scale of [B, 1, t].

[0152] In order to verify the method of the embodiment, the method of the embodiment is compared with Wiener, SEGAN, SERGAN, TSN, MetricGAN, Wave-U-Net, WaveCRN, DCCRN and TSEGAN models on the same data set to verify the effect of the embodiment on speech enhancement.

[0153] The experimental data set is a public data set, and the clean speech comes from the Voice Bank corpus (V. Christophe, Y. Junichi, and K. Simon, "The voice bank corpus: Design, collection and data analysis of a large regional accent speech database," in 2013 International Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA / CASLRE). IEEE, 2013, pp. 1-4.), and the noise comes from the DEMAND corpus (J. Thiemann, N. Ito, and E. Vincent, "The diverse environments multichannel acoustic noise database (DEMAND): A database of multichannel environmental noise recordings," in Meetings on Acoustics, vol. 19, 2013, pp. 35-81.). There are 12,396 speech data from 30 people, of which 11,572 speech from 28 people are used as the training set, and 824 speech from the other 2 people are used as the test set. The speech signal-to-noise ratio of the training set is 0, 5, 10 and 15 dB, and the speech signal-to-noise ratio of the test set is 2.5, 7.5, 12.5 and 17.5 dB.

[0154] The embodiment uses PESQ (Perceptual Evaluation of Speech Quality), STOI (Short-Time Objective Intelligibility) commonly used in speech enhancement tasks, and subjective score prediction CSIG, CBAK and COVL for evaluating signal distortion, background noise and overall effect, respectively, as indexes. The experimental results are shown in Table 1. As can be seen from Table 1, the speech enhancement method based on scale-sensitive signal-to-noise ratio loss and double-branch complex convolution recurrent neural network provided in the embodiment can obtain enhanced speech with better speech perceptual quality and intelligibility.

[0155] Table 1 Comparison of evaluation indexes of different models

[0156] Model PESQ STOI CSIG CBAK COVL NOISY 1.97 0.921 3.35 2.44 2.63 Wiener 2.22 / 3.23 2.68 2.67 SEGAN 2.16 0.925 3.48 2.94 2.80 SERGAN 2.51 0.938 3.79 3.24 3.14 TSN 2.68 0.925 3.96 2.94 3.32 MetricGAN 2.86 / 3.99 3.18 3.42 Wave-U-Net 2.62 / 3.94 3.37 3.29 WaveCRN 2.64 / 3.94 3.37 3.29 DCCRN 2.75 0.931 3.82 2.72 3.28 TSEGAN 2.52 / 3.67 3.27 3.08 Model in this paper 2.98 0.940 3.93 3.51 3.45

[0157] Embodiment two:

[0158] The embodiment of the application provides a speech enhancement device, the device comprises:

[0159] The speech acquisition module is configured to acquire a time-domain waveform of the speech to be enhanced.

[0160] The speech enhancement module is configured to input the speech to be enhanced into the trained double-branch complex convolution recurrent neural network to acquire time-domain waveforms of the speech branch and the noise branch.

[0161] The speech output module is configured to output the time-domain waveform of the speech branch as an enhanced result.

[0162] The training of the double-branch complex convolution recurrent neural network comprises:

[0163] The data acquisition module is configured to acquire time-domain waveforms of a preset number of noisy speeches, and time-domain waveforms of pure noise speeches and clean speeches corresponding to the noisy speeches.

[0164] The network training module is configured to construct a double-branch complex convolution recurrent neural network, input the time-domain waveforms of the noisy speeches into the constructed double-branch complex convolution recurrent neural network, and acquire time-domain waveforms of the speech branch and the noise branch.

[0165] The loss calculation module is configured to calculate a first scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the pure noise speeches and the noise branch corresponding to the noisy speeches, calculate a second scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the clean speeches and the speech branch corresponding to the noisy speeches, and sum the first scale-sensitive signal-to-noise ratio loss and the second scale-sensitive signal-to-noise ratio loss to obtain a total loss.

[0166] A network optimization module is configured to optimize the double-branch complex convolution recurrent neural network by using a gradient descent algorithm according to a total loss until a preset training termination condition is reached.

[0167] Embodiment three:

[0168] Based on embodiment one, the embodiment of the present application provides an electronic device, comprising a processor and a storage medium.

[0169] The storage medium is configured to store instructions.

[0170] The processor is configured to operate according to the instructions to perform steps of the above method.

[0171] Embodiment four:

[0172] Based on embodiment one, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement steps of the above method.

[0173] Those skilled in the art should understand that embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0174] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0175] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0176] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0177] The above description is only the preferred embodiment of the present application, it should be pointed out that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A speech enhancement method, characterized by, The method comprises: acquiring a time-domain waveform of a to-be-enhanced voice; inputting the to-be-enhanced voice into a trained double-branch complex convolution recurrent neural network to acquire time-domain waveforms of a voice branch and a noise branch; outputting the time-domain waveform of the voice branch as an enhancement result; wherein the training of the double-branch complex convolution recurrent neural network comprises: acquiring time-domain waveforms of a preset number of noisy voices, and time-domain waveforms of pure noise voices and clean voices corresponding to the noisy voices; constructing a double-branch complex convolution recurrent neural network, inputting the time-domain waveforms of the noisy voices into the constructed double-branch complex convolution recurrent neural network to acquire time-domain waveforms of the voice branch and the noise branch; calculating a first scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the pure noise voices corresponding to the noisy voices and the noise branch, and calculating a second scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the clean voices corresponding to the noisy voices and the voice branch, and summing the first scale-sensitive signal-to-noise ratio loss and the second scale-sensitive signal-to-noise ratio loss to acquire a total loss; optimizing the double-branch complex convolution recurrent neural network by using a gradient descent algorithm according to the total loss until a preset training termination condition is reached; wherein the double-branch complex convolution recurrent neural network comprises a differentiable Fourier transform module, a complex convolution module group of the voice branch and the noise branch, a first bridge module group, a complex long short-term memory module group of the voice branch and the noise branch, a second bridge module group, a complex deconvolution module group of the voice branch and the noise branch, and a differentiable inverse Fourier transform module; the differentiable Fourier transform module is realized by a convolution layer and is used to convert a time-domain waveform of an input voice into a complex time-frequency spectrum feature; each of the complex convolution module groups of the voice branch and the noise branch comprises N layers of complex convolution modules connected in a head-to-tail manner, the first bridge module group comprises N layers of bridge modules, and each bridge module of the first bridge module group is used to connect the corresponding layers of complex convolution modules of the voice branch and the noise branch; the complex convolution module groups of the voice branch and the noise branch are respectively used to process the complex time-frequency spectrum feature to acquire a first complex feature of the voice branch and the noise branch; each of the complex long short-term memory module groups of the voice branch and the noise branch comprises M layers of complex long short-term memory modules connected in a head-to-tail manner, and the complex long short-term memory modules of the voice branch and the noise branch are respectively used to process the first complex feature of the voice branch and the noise branch to acquire a second complex feature of the voice branch and the noise branch; each of the complex deconvolution module groups of the voice branch and the noise branch comprises N layers of complex deconvolution modules connected in a head-to-tail manner, the second bridge module group comprises N layers of bridge modules, and each bridge module of the second bridge module group is used to connect the corresponding layers of complex deconvolution modules of the voice branch and the noise branch; the kth layer of complex deconvolution module of the voice branch and the noise branch is connected with the N-k+1th layer of complex convolution module of the voice branch and the noise branch; and the complex deconvolution module groups of the voice branch and the noise branch are respectively used to process the second complex feature of the voice branch and the noise branch to acquire a third complex feature of the voice branch and the noise branch. The complex multiplication operation results of the speech branch and the noise branch are obtained by performing complex multiplication operation on the complex time-frequency spectrum features of the speech branch and the noise branch, respectively; The differentiable inverse Fourier transform module is implemented by a convolution layer, and is configured to convert the complex multiplication operation results of the speech branch and the noise branch into time-domain waveforms of the input speech in the speech branch and the noise branch.

2. The speech enhancement method of claim 1, wherein, The bridge module comprises two complex convolution modules, the expressions of the two complex convolution modules are Bridge s , Bridge n : Bridge s = cConvModule s (X n )+ X s Bridge n = cConvModule n (X s )+ X n where X s , X n are the inputs of the bridge module at the speech branch and the noise branch; cConvModule s , cConvModule n is a complex convolution module with inputs X n , X s ; Bridge s , Bridge n are the outputs of the bridge module at the speech branch and the noise branch.

3. The speech enhancement method according to claim 1 or 2, characterized in that, The complex convolution module comprises a complex convolution layer, a complex batch normalization layer and a complex PReLU layer connected in sequence; An expression S1 of the complex convolution layer is: In the formula, X, S1 are the input and output of a complex convolution layer, X r , X i are the real part and the imaginary part of the input X, W is the convolution kernel of the complex convolution layer, W r , W i are the real part and the imaginary part of the convolution kernel W, is a complex convolution operation, and * is a convolution operation. An expression S2 of the complex batch normalization layer is: S2 = cBN(S1) = BN(S 1r )+iBN(S 1i ) where S2 is the output of a complex batch normalization layer, cBN is a complex batch normalization operation, BN is a batch normalization operation, S 1r , S 1i is the real and imaginary parts of S1; An expression S3 of the complex PReLU layer is: S3 = cPReLU(S2) = PReLU(S 2r )+iPReLU(S 2i ) In the formula, S3 is the output of the complex PReLU layer, cPReLU is the complex PReLU operation, PReLU is the PReLU operation, S 2r is the real part and S 2i is the imaginary part of S2.

4. The speech enhancement method of claim 1, wherein, An expression S4 of the complex long short-term memory module is: S4 = cLSTM(X') = (out rr -out ii ) + i(out ri -out ir ) out rr = LSTM r (X′ r ) out ir = LSTM r (X′ i ) out ri = LSTM i (X′ r ) out ii = LSTM i (X′ i ) In the formula, X', S4 are the input and output of the complex long short-time memory module, X' r , X' i are the real part and imaginary part of the input X', cLSTM is the complex long short-time memory operation, LSTM r , LSTM i is the long short-time memory operation of the real part and imaginary part.

5. The speech enhancement method of claim 1, wherein, The complex deconvolution module comprises a complex deconvolution layer, a complex batch normalization layer and a complex PReLU layer connected in sequence; An expression S5 of the complex convolution layer is: where X", S5are the input and output of the complex deconvolution layer, X" r , i are the real and imaginary parts of the input X", W" is the deconvolution kernel of the complex convolution layer, W r , i are the real and imaginary parts of the deconvolution kernel W", is the complex deconvolution operation, is the deconvolution operation; An expression S6 of the complex batch normalization layer is: S6 = cBN(S5) = BN(S 5r )+iBN(S 5i ) where S6 is the output of the complex batch normalization layer, cBN is the complex batch normalization operation, BN is the batch normalization operation, S 5r , S 5i are the real and imaginary parts of S5; An expression S7 of the complex PReLU layer is: S7 = cPReLU(S6) = PReLU(S 6r )+iPReLU(S 6i ) In the formula, S7 is the output of the complex PReLU layer, cPReLU is the complex PReLU operation, PReLU is the PReLU operation, S6 is the output of the PReLU layer, and S 6r , S 6i are the real part and the imaginary part of S6, respectively.

6. The speech enhancement method of claim 1, wherein, An expression SA_SNRLoss of the total loss is: wherein is the first scale sensitive signal-to-noise ratio loss and the second scale sensitive signal-to-noise ratio loss, s1, is the time-domain waveform of the clean speech and speech branch corresponding to the noisy speech, s2, is the time-domain waveform of the clean speech and speech branch corresponding to the noisy speech, |·| is an L1 norm operator, |·| 2 is an L2 norm operator, is s m dot product operation of s1and s2, m = 1, 2.

7. A speech enhancement device, characterized by The apparatus comprises: A speech acquisition module configured to acquire a time-domain waveform of a speech to be enhanced; A speech enhancement module configured to input the speech to be enhanced into the trained double-branch complex convolution recurrent neural network to obtain time-domain waveforms of the speech branch and the noise branch; A speech output module configured to output the time-domain waveform of the speech branch as an enhanced result. The training of the double-branch complex convolution recurrent neural network comprises: A data acquisition module configured to acquire time-domain waveforms of a preset number of noisy speeches, and time-domain waveforms of pure noise speeches and clean speeches corresponding to the noisy speeches; A network training module configured to construct a double-branch complex convolution recurrent neural network, input the time-domain waveforms of the noisy speeches into the constructed double-branch complex convolution recurrent neural network, and obtain time-domain waveforms of the speech branch and the noise branch; A loss calculation module configured to calculate a first scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the pure noise speeches and the noise branch corresponding to the noisy speeches, calculate a second scale-sensitive signal-to-noise ratio loss according to the time-domain waveforms of the clean speeches and the speech branch corresponding to the noisy speeches, and sum the first scale-sensitive signal-to-noise ratio loss and the second scale-sensitive signal-to-noise ratio loss to obtain a total loss; A network optimization module configured to optimize the double-branch complex convolution recurrent neural network according to the total loss by using a gradient descent algorithm until a preset training termination condition is reached. The double-branch complex convolution recurrent neural network comprises a differentiable Fourier transform module, a complex convolution module group of the speech branch and the noise branch, a first bridge module group, a complex long short-term memory module group of the speech branch and the noise branch, a second bridge module group, a complex deconvolution module group of the speech branch and the noise branch, and a differentiable inverse Fourier transform module. The differentiable Fourier transform module is implemented by a convolution layer, and is configured to convert the time-domain waveform of the input speech into complex time-frequency spectrum features. The complex convolution module group of the speech branch and the noise branch each includes N layers of complex convolution modules connected in a head-to-tail manner, the first bridge module group includes N layers of bridge modules, and each layer of the bridge modules of the first bridge module group is configured to connect the corresponding layer of complex convolution modules of the speech branch and the noise branch; the complex convolution module group of the speech branch and the noise branch is respectively configured to process the complex time-frequency spectrum feature to obtain a first complex feature of the speech branch and the noise branch; The complex long short-term memory module group of the speech branch and the noise branch each includes M layers of complex long short-term memory modules connected in a head-to-tail manner, and the complex long short-term memory module of the speech branch and the noise branch is respectively configured to process the first complex feature of the speech branch and the noise branch to obtain a second complex feature of the speech branch and the noise branch; The complex deconvolution module group of the speech branch and the noise branch each includes N layers of complex deconvolution modules connected in a head-to-tail manner, the second bridge module group includes N layers of bridge modules, and each layer of the bridge modules of the second bridge module group is configured to connect the corresponding layer of complex deconvolution modules of the speech branch and the noise branch; the kth layer of complex deconvolution modules of the speech branch and the noise branch is connected to the N-k+1th layer of complex convolution modules of the speech branch and the noise branch; the complex deconvolution module group of the speech branch and the noise branch is respectively configured to process the second complex feature of the speech branch and the noise branch to obtain a third complex feature of the speech branch and the noise branch; The complex time-frequency spectrum feature is subjected to a complex multiplication operation with the third complex feature of the speech branch and the noise branch to obtain a complex multiplication operation result of the speech branch and the noise branch; The differentiable inverse Fourier transform module is implemented by a convolution layer and is configured to convert the complex multiplication operation result of the speech branch and the noise branch into a time-domain waveform of the input speech in the speech branch and the noise branch.

8. An electronic device, comprising: The computer program product comprises a processor and a storage medium; The storage medium is configured to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-6.

Images