Single-channel speech enhancement method and device based on mean reversal Schrodinger bridge

Abstract

The invention discloses a single-channel speech enhancement method and device based on a mean-reversal Schrodinger bridge, and the method comprises the steps: forming a sample pair set through a pure speech sample and a noisy speech sample, carrying out the preprocessing and Fourier transformation of the speech samples, and obtaining a frequency spectrum complex matrix pair; constructing discriminant models, estimating a frequency spectrum complex matrix of the pure voice, adjusting model parameters according to the difference between the frequency spectrum complex matrix of the pure voice and the frequency spectrum complex matrix of the real pure voice, and taking the discriminant model corresponding to the optimal parameter as a target discriminant model; constructing a score model, estimating a reverse optimal migration score in a reverse generation process, adjusting model parameters according to the difference between the reverse optimal migration score and a real reverse migration score, and taking the score model corresponding to the optimal parameter as a target score model; and giving noisy voice, inverting the reverse optimal offset score of the Schrodinger bridge by using the trained target score model parameterization mean value, and carrying out a reverse generation process to generate pure voice. The method is lower in calculation cost and better in speech enhancement effect.

Description

Technical Field

[0001] The present invention relates to the field of speech enhancement in intelligent speech technology, and in particular to a single-channel speech enhancement method and device based on a mean-reversal Schrödinger bridge. Background Art

[0002] Speech Enhancement (SE) technology aims to estimate a pure waveform from a noisy speech waveform and improve the naturalness and intelligibility of speech. It is widely used in speech recognition systems, online conferencing, smart homes and other fields.

[0003] Generally speaking, speech enhancement methods can be divided into two categories: discriminative and generative. The discriminative method can effectively eliminate noise in the speech waveform by directly modeling the deterministic mapping between noisy speech and clean speech, but it will cause problems with speech distortion and over-denoising. The generative method implicitly or explicitly learns the potential data distribution of clean speech. Its output is more natural and understandable, and it shows stronger generalization ability for various noise patterns. In particular, the generative method based on the diffusion model regards the speech enhancement task as a conditional generation process or a probability distribution transmission process, which effectively improves the perceptual quality of the enhanced speech. However, because the inverse generation process requires multiple iterations, the computational cost is high and it cannot meet the needs of real-time speech processing.

[0004] The latest research combines the advantages of the above two methods to develop a hybrid speech enhancement method, that is, cascading a discriminant model and a diffusion model. The discriminant model of this type of method is used to output the initial predicted waveform of pure speech, and then the diffusion model repairs the fuzziness and distortion of the initial prediction, which reduces the number of iterations of the diffusion model to a certain extent and produces pure speech with higher perceived quality. However, there are still some problems with this hybrid method. On the one hand, the inverse generation process of the diffusion model still requires at least 20 neural network calls, and the computational cost is still high. On the other hand, due to the limited generalization ability of the pre-discriminant model, the generalization ability is poor in complex acoustic scenarios. Summary of the invention

[0005] In view of the shortcomings in the prior art, the present invention provides a single-channel speech enhancement method and device based on a mean-reversal Schrödinger bridge.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] Single-channel speech enhancement method based on mean-reversal Schrödinger bridge:

[0008] The clean speech samples and the noisy speech samples are mixed at different signal-to-noise ratios to obtain noisy speech samples, and the clean speech samples and the noisy speech samples form a speech sample pair set;

[0009] Preprocessing the speech samples in the speech sample pair set, and then performing Fourier transformation to obtain a frequency spectrum complex matrix pair;

[0010] A discriminant model is constructed, with the spectral complex matrix of the noisy speech as input and the spectral complex matrix of the estimated clean speech as output; based on the difference between the spectral complex matrix of the estimated clean speech and the spectral complex matrix of the real clean speech, the parameters of the discriminant model are adjusted, and when the discriminant model meets the preset training requirements, the discriminant model corresponding to the optimal parameters is used as the target discriminant model;

[0011] Constructing a score model, using the score model to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, so as to estimate the inverse optimal offset score in the inverse generation process; adjusting the parameters of the score model based on the difference between the estimated inverse optimal offset score and the true inverse optimal offset score, and when the score model meets the preset training requirements, using the score model corresponding to the optimal parameters as the target score model;

[0012] Given noisy speech, the trained target score model is used to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, and the inverse generation process is performed to generate clean speech.

[0013] Furthermore, the process of obtaining the spectral complex matrix pair is as follows: for any speech sample pair (x, y), randomly intercept sub-segments of x and y, and count the maximum value of the sampled value in y, and use the maximum value to normalize x and y respectively to obtain x′ and y′, and obtain the spectral complex matrix X corresponding to x′ and y′ through Fourier transform. 0 , Y 0 , and then X 0 and Y 0 Perform amplitude compression to obtain a complex spectrum matrix pair (X, Y).

[0014] Furthermore, the training process of the discriminative model is:

[0015] Input the complex matrix Y of the spectrum of noisy speech into the discriminant model D θ , output the estimated value of the complex matrix of the spectrum of the pure speech

[0016] Constructing reconstruction loss function based on pure speech where L(·) represents the difference between the two; to minimize the loss function For the training target, perform gradient back propagation and update the parameters θ;

[0017] When the preset training requirements are met, stop training; select the optimal parameter θ during the training process *The corresponding discriminant model is used as the target discriminant model.

[0018] Furthermore, the process of obtaining the mean-reversal Schrödinger bridge is:

[0019] For a spectral complex matrix pair (X,Y), assume that the posterior mean of X conditioned on Y is the variable When Y is given, the variable From a conditional posterior distribution It is observed that the target discriminant model is used Approximate the conditional posterior distribution, that is

[0020] The mean reversal process at a specific time The state sample is represented as sample Obey marginal Gaussian distribution in, represents a uniform distribution on the interval [0,1],

[0021] Define a Schrödinger bridge process that bridges the noisy speech distribution and the clean speech distribution, and represent the state sample of the Schrödinger bridge process at time t as X t , sample X t Obeying the marginal Gaussian distribution q(X t |X0,X1), and X0=X, X1=Y; at time t, assuming X t by As a conditional representation, the edge Gaussian distribution of the Schrödinger bridge is derived as:

[0022]

[0023]

[0024] in, yes Point estimate of

[0025] The Schrödinger bridge process that introduces the mean reversal process is called the mean reversal Schrödinger bridge, which replaces the boundary condition X0=X of the Schrödinger bridge at time t with

[0026] Furthermore, the mean reversal process is a Schrodinger bridge process that bridges the posterior mean distribution and the pure speech distribution.

[0027] Furthermore, at time t, the marginal Gaussian distribution of the mean reversal process is:

[0028]

[0029] in, represents the mean of the marginal distribution of the mean reversal process at time t, represents the variance of the marginal distribution of the mean reversal process at time t, α t and σ t is the noise schedule of the mean reversal process, the signal-to-noise ratio

[0030] Furthermore, the mean-reversing Schrödinger bridge is described by the following two stochastic differential equations:

[0031]

[0032] Where f is the shift coefficient, g is the diffusion coefficient, W and is a standard Wiener process, Ψ t and is the Schrödinger factor, the fraction and represents the positive optimal shift score and the reverse optimal shift score;

[0033] At time t, the edge Gaussian distribution of the mean-reversed Schrödinger bridge is:

[0034]

[0035] in, represents the mean of the edge distribution of the mean-reversed Schrödinger bridge at time t, represents the variance of the mean-reversed Schrödinger bridge edge distribution at time t, and is the noise schedule of the mean-reversal Schrödinger bridge, which is determined by f and g in the above stochastic differential equation, and the signal-to-noise ratio

[0036] Furthermore, the training process of the score model is:

[0037] Input the complex matrix Y of the spectrum of the noisy speech into the target discriminant model Output

[0038] Random sampling time According to the marginal Gaussian distribution of the mean reversal process at time t Calculate the mean of the distribution and get the point estimate at time t Next, the edge Gaussian distribution of the Schrödinger bridge is inverted according to the mean at time t Sampling to get the sample at time t in is random noise; finally, the time t, sample X t Input score model Output The actual reverse optimal offset score at this time for The estimated inverse optimal shift score is

[0039] The score model parameters are adjusted based on the difference between the estimated reverse optimal offset score and the actual reverse optimal offset score. The difference between the two is reflected in the difference between X0 and Specifically: define the data prediction loss function To minimize the loss function For the training target, perform gradient back propagation and update parameters

[0040] When the preset training requirements are met, stop training; select the optimal parameters during training The corresponding score model is used as the target score model.

[0041] Furthermore, the reverse generation process is specifically as follows:

[0042] Set the time t j ,sample Input the target score model and get

[0043] pass Estimating the inverse optimal offset score Solve inverse stochastic differential equations calculate

[0044] Let j = j-1. If j>0, repeat the above process. If j=0, stop the reverse generation. The sample at time t0 is The complex matrix of the spectrum of the clean speech after the noisy speech enhancement is then subjected to inverse amplitude compression, inverse Fourier transform and inverse normalization to finally obtain the waveform of the clean speech.

[0045] A single-channel speech enhancement device based on a mean-reversal Schrödinger bridge, comprising:

[0046] Construct a sample pair module, which consists of a speech sample pair set consisting of a clean speech sample and a noisy speech sample;

[0047] A sample processing module performs preprocessing and Fourier transformation on the speech samples in the speech sample pair set to obtain a spectrum complex matrix pair;

[0048] A discriminant model building module is used to build a discriminant model, estimate the spectral complex matrix of clean speech, and adjust the parameters of the model based on the difference between the estimated spectral complex matrix of clean speech and the spectral complex matrix of real clean speech. When the model meets the preset training requirements, the discriminant model corresponding to the optimal parameters is used as the target discriminant model;

[0049] A score model construction module is used to construct a score model, estimate the reverse optimal offset score during the reverse generation process, adjust the parameters of the model based on the difference between the estimated reverse optimal offset score and the true reverse optimal offset score, and when the model meets the preset training requirements, use the score model corresponding to the optimal parameters as the target score model;

[0050] The clean speech generation module, for a given noisy speech, uses the trained target score model to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, performs a reverse generation process, and generates clean speech.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) The present invention proposes a single-channel speech enhancement framework based on mean-reversed Schrödinger bridge, defines the speech enhancement task as the optimal transmission problem between the noisy speech distribution and the clean speech distribution, and uses the idea of ​​diffusion model to solve the corresponding Schrödinger bridge problem to achieve clean speech generation. Compared with the existing single-channel speech enhancement method based on diffusion model on public data sets, the present invention has more advanced performance, significantly reduces the number of neural network calls, and has lower computational cost.

[0053] (2) The present invention improves the traditional Schrödinger bridge and proposes a mean-reversed Schrödinger bridge. The discriminant model is used to introduce the posterior mean (the posterior mean of the clean speech conditioned by the noisy speech) information as a guide for the generation process. Unlike the existing hybrid method, the present invention only approximates the sampling marginal Gaussian distribution based on the discriminant model to calculate the inverse optimal offset score when training the score model, thereby avoiding the problem of limited generalization ability caused by the pre-discriminant model in the hybrid method. Specifically, the present invention designs a mean reversal process to simulate the conversion process from the posterior mean to the pure speech; then, by deriving the approximate form of the Gaussian marginal distribution, the mean reversal process is introduced in the Schrödinger bridge, and the posterior mean is used as a guide for the reverse generation process; in the early stage of the reverse generation process, the mean reversal process is biased towards the posterior mean samples, and the mean reversal Schrödinger bridge process samples are located outside the data manifold. Under the guidance of the posterior mean, the complexity of the neural network estimating the reverse optimal offset score can be reduced, and the ability to retain high-frequency information of the speech can be improved; and in the later stage of the generation process, the mean reversal process gradually biases towards the pure speech samples, and the mean reversal Schrödinger bridge process samples are located in the data manifold, and it is converted into a generation process for the pure speech samples, and the low-frequency information of the speech is gradually restored, and finally the optimal conversion of noisy speech and pure speech under the guidance of the posterior mean is achieved, and the speech enhancement effect is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 This is a flow chart of the single-channel speech enhancement based on the mean-reversal Schrödinger bridge described in the present invention. DETAILED DESCRIPTION

[0056] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0057] like Figure 1 As shown, the single-channel speech enhancement method based on mean-reversed Schrödinger bridge provided in this embodiment includes the following steps:

[0058] S1, constructing sample pairs: the clean speech samples and the noisy speech samples are mixed at different signal-to-noise ratios to obtain noisy speech samples, and the clean speech samples and the noisy speech samples are combined into a speech sample pair set.

[0059] In this embodiment, in a computer, VoiceBank+DEMAND speech enhancement benchmark data is obtained from the Internet, and the VoiceBank+DEMAND speech enhancement benchmark data set is used to directly obtain a pre-constructed speech sample pair set D xy ={(x (i) ,y (i) )|x (i) ∈D x ,y (i) ∈D y ,i=1,2,...,n}, where (x (i) ,y (i) ) is the i-th group of clean speech and noisy speech sample pairs, x (i) and (i) Represent the original waveform of pure speech and the original waveform of noisy speech respectively, D x ={x (1) ,x (2) ,...,x (n)} is a pure speech set, D y ={y (1) ,y (2) ,...,y (n)} is a noisy speech set, i is a sample index, and n is the total number of sample pairs. In this dataset, the clean speech comes from the VoiceBank corpus, and the noisy speech comes from the DEMAND corpus, and is mixed according to the signal-to-noise ratio of 0db, 5db, 10db, and 15db.

[0060] The sample pair construction method is not limited to using pre-constructed sample pairs from public datasets. It can also use clean speech samples from clean speech datasets such as VoiceBank, TIMIT, and WJS0, and noisy speech from noisy speech datasets such as DEMAND and WHAM!, and mix them according to a specific signal-to-noise ratio to construct speech sample pairs.

[0061] S2, speech preprocessing: After preprocessing the speech samples in the speech sample pair set, the corresponding spectrum complex matrix pairs are obtained through Fourier transform.

[0062] In this embodiment, when preprocessing any speech sample pair (x, y)∈D xy When , first randomly intercept sub-segments of x and y, and count the maximum value y of the sampled value in y max , use the maximum value to normalize x and y respectively, and get x′=x / y max , y′=y / y max ; Then, the spectrum complex matrix X corresponding to x′ and y′ is obtained by Fourier transform 0 , Y 0 ; Then, for X 0 and Y 0 Perform amplitude compression to obtain the final spectrum complex matrix pair (X, Y), where where |·| represents the modulus, ∠(·) represents the phase angle, and β s is the scaling factor, β m is an exponential factor. In this embodiment, β s =0.33, β m =0.5. In this embodiment, the original waveform x and the complex spectrum matrix X can both represent pure speech, and the original waveform y and the complex spectrum matrix Y can both represent noisy speech.

[0063] S3, discriminant model training: Construct a discriminant model for estimating the posterior mean of clean speech conditioned on noisy speech. The model takes the spectral complex matrix of noisy speech as input and the spectral complex matrix of estimated clean speech as output. According to the difference between the spectral complex matrix of estimated clean speech and the spectral complex matrix of real clean speech, the parameters of the discriminant model are adjusted. When the discriminant model meets the preset training requirements, the discriminant model corresponding to the optimal parameters is used as the target discriminant model.

[0064] In this embodiment, a discriminant model D is first constructed. θ , train the model as follows:

[0065] (1) Select a sample pair (x, y)∈D xy , preprocess according to S2 to obtain the spectral complex matrix pair (X,Y);

[0066] (2) Input the complex matrix Y of the spectrum of the noisy speech into D θ , output the estimated value of the complex matrix of the spectrum of the pure speech

[0067] (3) Constructing a reconstruction loss function based on pure speech Where L(·) represents the difference between the two, and the mean square error (MSE) can usually be used; to minimize the loss function For the training target, perform gradient back propagation and update the parameters θ;

[0068] (4) Repeat (1) to (3) until the training requirements such as the maximum number of iterations are met, then stop training; select the optimal parameter θ during the training process * The corresponding discriminant model is used as the target discriminant model; the target discriminant model is only used for the training of the score model in S4 and is not used for the clean speech generation in S5.

[0069] S4, fractional model training: construct a fractional model for matching the inverse optimal offset score of the mean-reversed Schrödinger bridge. Use the fractional model to parameterize the inverse optimal offset score of the mean-reversed Schrödinger bridge to estimate the inverse optimal offset score during the inverse generation process. According to the difference between the estimated inverse optimal offset score and the true inverse optimal offset score, adjust the parameters of the fractional model. When the preset training requirements are met, the fractional model corresponding to the optimal parameters is used as the target fractional model.

[0070] In this embodiment, the mean-reversed Schrödinger bridge is a special Schrödinger bridge, which improves the estimation capability of high-frequency information of pure speech and reduces the difficulty of score matching during training by introducing posterior mean information.

[0071] Specifically, for a specific spectral complex matrix pair (X, Y), first assume that the posterior mean of X conditioned on Y is the variable When Y is given, the variable From a conditional posterior distribution This embodiment uses the target discriminant model obtained by S3 training Approximate the conditional posterior distribution, that is

[0072] Next, we define a Schrödinger bridge process that bridges the posterior mean distribution and the pure speech distribution, called the mean reversal process. The state sample is represented as sample Obey marginal Gaussian distribution in represents a uniform distribution on the interval [0,1], At time t, the marginal Gaussian distribution of the mean reversal process can be specifically expressed as:

[0073]

[0074] in, represents the mean of the marginal distribution of the mean reversal process at time t, It represents the variance of the marginal distribution of the mean reversal process at time t, α t and σ t is the noise schedule of the mean reversal process, the signal-to-noise ratio In the early stage of the reverse generation process, the mean reversal process is biased towards the posterior mean samples; in the later stage, the mean reversal process gradually biases towards the pure speech samples, achieving the optimal conversion between the posterior mean and the pure speech.

[0075] Finally, a Schrödinger bridge process is defined to bridge the noisy speech distribution and the clean speech distribution. The state sample of the Schrödinger bridge process at time t is represented as X t , sample X t Obeying the marginal Gaussian distribution q(X t |X0,X1), and X0=X, X1=Y. The Schrödinger bridge can be described by the following two stochastic differential equations:

[0076]

[0077] Where f is the shift coefficient, g is the diffusion coefficient, W and is a standard Wiener process, Ψ t and is the Schrödinger factor, the fraction and Represents the forward optimal shift score and the reverse optimal shift score.

[0078] In order to introduce the mean reversal process into the Schrödinger bridge process, at time t, assume that X t by As a conditional representation, the approximate edge Gaussian distribution of the Schrödinger bridge is derived as:

[0079]

[0080] in, yes The point estimate of can usually be expressed as the distribution mean. Therefore, the Schrödinger bridge process that introduces the mean reversal process is called the mean reversal Schrödinger bridge, which replaces the boundary condition X0=X of the traditional Schrödinger bridge at time t with From the marginal Gaussian distribution X is obtained by approximate sampling t At time t, the edge Gaussian distribution of the mean-reversal Schrödinger bridge can be specifically expressed as:

[0081]

[0082] in, represents the mean of the edge distribution of the mean-reversed Schrödinger bridge at time t, It represents the variance of the mean-reversal Schrödinger bridge edge distribution at time t, and is the noise dispatch of the mean-reversal Schrödinger bridge, which is determined by f and g in the above stochastic differential equation (the determination process is the prior art), and the signal-to-noise ratio

[0083] The reverse generation process of the mean-reversed Schrödinger bridge refers to the process of transforming from 1 to 0 at time t, corresponding to sample X t The mean-reversed Schrödinger bridge gradually changes from noisy speech to clean speech. The characteristics of the mean-reversed Schrödinger bridge are: due to the approximate replacement of boundary conditions, in the early stage of the reverse generation process, the mean-reversed process is biased towards the posterior mean sample, and the mean-reversed Schrödinger bridge process uses the posterior mean as a condition to maximize the restoration of the high-frequency information of the clean speech from the noisy speech; in the later stage, the mean-reversed process gradually biases towards the clean speech sample, and the mean-reversed Schrödinger bridge process gradually restores the low-frequency information, and finally achieves the optimal conversion of noisy speech and clean speech under the guidance of the posterior mean.

[0084] During the fractional model training process, since the boundary conditions X0 and X1 are known, the reverse generation process of the mean-reversed Schrödinger bridge is also known. At this time, the reverse optimal offset score at time t can be expressed as Inverse Optimal Offset Score It describes the optimal path for the sample to be transferred from the noisy speech distribution to the clean speech distribution, indicating the direction of the gradual transition from the noisy speech sample to the clean speech sample. In the actual speech enhancement process, only the boundary condition X1 is known, and the inverse optimal offset score at time t can be parameterized by constructing a fractional model The training goal of the score model is to ensure that the estimated inverse optimal offset score can accurately guide the sample to evolve from a noisy state to a pure state, thereby achieving the effect of speech enhancement.

[0085] In this embodiment, when training the score model, first construct a score model This model usually adopts a U-Net structure with residual connections. Inverse optimal offset fraction for parameterized mean-reversal Schrödinger bridge When , the model can usually be used to predict the score and directly match the real score, or the real score can be indirectly matched by predicting noise, predicting data, etc. This embodiment uses the training method of indirectly matching the real score by predicting data, that is: Predict X0 to estimate the inverse optimal offset score, expressed as

[0086] Specifically, this embodiment trains the score model according to the following process:

[0087] (1) Select a sample pair (x, y)∈D xy , preprocess according to S2 to obtain the spectral complex matrix pair (X,Y);

[0088] (2) Input the complex matrix Y of the spectrum of the noisy speech into the target discriminant model trained by S3 Output

[0089] (3) Random sampling time According to the marginal Gaussian distribution of the above mean reversal process at time t Calculate the mean of the distribution and get the point estimate at time t Next, the edge Gaussian distribution of the Schrödinger bridge is inverted according to the mean at time t Sampling to get the sample at time t in is random noise; finally, the time t, sample X t Input score model Output The actual reverse optimal offset score at this time for The estimated inverse optimal shift score is

[0090] (4) Based on the difference between the estimated inverse optimal shift score and the actual inverse optimal shift score (the difference between the two is reflected in the difference between X0 and difference), adjust the score model parameters: define the data prediction loss function Minimizing this loss function can improve the accuracy of estimating the inverse optimal offset score, where L(·) represents the difference between the two, and the mean square error can usually be used; to minimize the loss function For the training target, perform gradient back propagation and update parameters

[0091] (5) Repeat (1) to (4) until the training requirements such as the maximum number of iterations are met, then stop training. Select the optimal parameters during training The corresponding score model is used as the target score model.

[0092] S5, clean speech generation: Given noisy speech, use the trained target score model to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, perform the reverse generation process, and generate clean speech. The initial state of the reverse generation process is noisy speech, and the final state obtained after multiple iterations is the estimated clean speech.

[0093] In this embodiment, given a noisy speech y tgt Referring to the preprocessing method of S2, the noisy speech is normalized, Fourier transformed, and amplitude compressed in sequence without intercepting the sub-segment, and the corresponding spectrum complex matrix Y is obtained. tgt . Let the reverse generated time list be [t0, t1, ..., t i , ..., t N ], where 0 <t i ≤1, t0=0.03, t N =1, i = 0, 1, ..., N, N is the number of iterations. Let j = N, the initial state Follow the following process to reverse the generation:

[0094] (1) Set time t j ,sample Input the target score model and get

[0095] (2) Through Estimating the inverse optimal offset score Solve inverse stochastic differential equations calculate A simple solution is: j-1 Sample The posterior probability distribution of Schrödinger bridge In known samples In the case of Replace X0, from Medium Sampling

[0096] (3) Let j = j-1. If j>0, repeat (1) to (2). If j = 0, stop the reverse generation. The sample at time t0 is The complex matrix of the spectrum of the clean speech after the noisy speech enhancement is subjected to the operations of inverse amplitude compression, inverse Fourier transform and inverse normalization to finally obtain the waveform of the clean speech.

[0097] In the above reverse generation process, since the score model is trained by indirectly matching the real score with the predicted data, the mean inversion Schrödinger bridge can be simulated by directly sampling the posterior probability distribution in (2) above. If the score model is trained by directly matching the real score with the score prediction or indirectly matching the real score with the predicted noise, the reverse generation process is similar to the above process and will not be described here.

[0098] In this embodiment, the VoiceBank+DEMAND and TIMIT+WHAM! datasets are used to train the discriminant model and the score model respectively to achieve speech enhancement. VoiceBank+DEMAND is a public speech enhancement benchmark dataset, and sample pairs have been pre-constructed. The TIMIT+WHAM! dataset is constructed by mixing the clean speech from TIMIT and the noisy speech from WHAM! with random signal-to-noise ratio speech according to the steps of S1 to generate noisy speech, and construct sample pairs; wherein the signal-to-noise ratio is uniformly sampled between -6db and 14db.

[0099] In this embodiment, the speech enhancement performance is compared with three mainstream diffusion model-based single-channel speech enhancement methods, namely CDiffuSE, SGMSE+, and StoRM, based on the VoiceBank+DEMAND dataset. The results are shown in Table 1.

[0100] Table 1 Comparison of speech enhancement performance with three mainstream methods on the VoiceBank+DEMAND dataset

[0101]

[0102] Among them, PESQ is an objective speech quality perception evaluation index, and the larger the value, the better; ESTOI is an objective short-term speech intelligibility evaluation index, and the larger the value, the better; SI-SDR is used to evaluate the degree of speech distortion, and the larger the value, the better; DNSMOS is a subjective speech quality perception evaluation index, and the larger the value, the better.

[0103] As shown in Table 1, compared with other methods, the method of this embodiment has improved in PESQ, ESTOI, SI-SDR, and DNSMOS, proving the advanced performance of this method. At the same time, this method only calls the neural network three times in the reverse generation process, and the calculation cost is significantly reduced.

[0104] In this embodiment, based on the TIMIT+WHAM! dataset and the VoiceBank+DEMAND dataset, a generalization comparison is performed with two mainstream diffusion model-based single-channel speech enhancement methods, SGMSE+ and StoRM. The results are shown in Table 2.

[0105] Table 2 Generalization comparison results with two mainstream methods on the TIMIT+WHAM! dataset

[0106]

[0107] WER is the word error rate of the recognized text in the downstream speech recognition task, and the lower the value, the better. Domain matching means that the method is trained on the TIMIT+WHAM! dataset and verified on the test set of the TIMIT+WHAM! dataset. Domain mismatch means that the method is trained on the VoiceBank+DEMAND dataset and verified on the test set of the TIMIT+WHAM! dataset.

[0108] As shown in Table 2, compared with other methods, the method of this embodiment achieves the best performance in both domain matching and domain mismatching cases, which proves the generalization ability of this method.

[0109] In this embodiment, an ablation experiment was performed based on the VoiceBank+DEMAND dataset, and the results are shown in Table 3.

[0110] Table 3 Ablation experiment results on VoiceBank+DEMAND dataset

[0111]

[0112] Among them, the second to fifth rows are improvements on the traditional Schrödinger bridge method. In the methods of the second to fourth rows, the discriminant model will exist in both the training process and the reverse generation process of the score model. However, in the fifth row method, that is, the method of this embodiment, the discriminant model only exists in the training process of the score model.

[0113] As shown in Table 3, the method of this embodiment achieves the highest objective speech perception quality (PESQ) and intelligibility (ESTOI) without the assistance of a discriminant model during the reverse generation process, and the other performances are also improved compared to the traditional Schrodinger bridge method.

[0114] Based on the same inventive concept, this embodiment also provides a single-channel speech enhancement device based on a mean-reversed Schrödinger bridge, comprising:

[0115] Construct a sample pair module, which consists of a speech sample pair set consisting of a clean speech sample and a noisy speech sample;

[0116] A sample processing module performs preprocessing and Fourier transformation on the speech samples in the speech sample pair set to obtain a spectrum complex matrix pair;

[0117] A discriminant model construction module is used to construct a discriminant model, taking the spectral complex matrix of noisy speech as input and the spectral complex matrix of estimated clean speech as output. Based on the difference between the spectral complex matrix of estimated clean speech and the spectral complex matrix of real clean speech, the parameters of the discriminant model are adjusted. When the discriminant model meets the preset training requirements, the discriminant model corresponding to the optimal parameters is used as the target discriminant model.

[0118] A score model construction module is used to construct a score model, parameterize the inverse optimal offset score of the mean-reversed Schrödinger bridge, estimate the inverse optimal offset score in the inverse generation process, adjust the parameters of the score model based on the difference between the estimated inverse optimal offset score and the true inverse optimal offset score, and when the score model meets the preset training requirements, use the score model corresponding to the optimal parameters as the target score model;

[0119] The clean speech generation module, for a given noisy speech, uses the trained target score model to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, performs a reverse generation process, and generates clean speech.

[0120] It should be noted that when the single-channel speech enhancement device based on the mean-reversed Schrödinger bridge provided in the above embodiment performs speech enhancement and generates pure speech, the division of the above-mentioned functional modules should be used as an example to illustrate that the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the single-channel speech enhancement device based on the mean-reversed Schrödinger bridge provided in the above embodiment belongs to the same concept as the single-channel speech enhancement method embodiment based on the mean-reversed Schrödinger bridge. The specific implementation process and effects are detailed in the single-channel speech enhancement method embodiment based on the mean-reversed Schrödinger bridge, which will not be repeated here.

[0121] Based on the same inventive concept, this embodiment also provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned single-channel speech enhancement method based on the mean-reversed Schrödinger bridge.

[0122] The computing device provided in this embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the single-channel speech enhancement method based on the mean-reversed Schrödinger bridge described in S1-S5 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0123] Based on the same inventive concept, this embodiment also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned single-channel speech enhancement method based on the mean-reversal Schrödinger bridge is implemented.

[0124] In this embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0125] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A single-channel speech enhancement method based on mean-reversal Schrödinger bridge, characterized in that: The clean speech samples and the noisy speech samples are mixed at different signal-to-noise ratios to obtain noisy speech samples, and the clean speech samples and the noisy speech samples form a speech sample pair set; Preprocessing the speech samples in the speech sample pair set, and then performing Fourier transformation to obtain a frequency spectrum complex matrix pair; Construct a discriminative model, taking the complex spectrum matrix of noisy speech as input and the complex spectrum matrix of estimated clean speech as output; Based on the difference between the estimated complex spectrum matrix of the clean speech and the complex spectrum matrix of the real clean speech, the parameters of the discriminant model are adjusted, and when the discriminant model meets the preset training requirements, the discriminant model corresponding to the optimal parameters is used as the target discriminant model; Constructing a score model, using the score model to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, so as to estimate the inverse optimal offset score in the inverse generation process; adjusting the parameters of the score model based on the difference between the estimated inverse optimal offset score and the true inverse optimal offset score, and when the score model meets the preset training requirements, using the score model corresponding to the optimal parameters as the target score model; Given noisy speech, the trained target score model is used to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, and the inverse generation process is performed to generate clean speech.

2. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 1, characterized in that: The process of obtaining the spectral complex matrix pair is as follows: for any speech sample pair (x, y), randomly intercept the sub-segments of x and y, and count the maximum value of the sample value in y, use the maximum value to normalize x and y respectively, and obtain x' and y', and obtain the spectral complex matrix X corresponding to x' and y' through Fourier transform 0 , Y 0 , and then X 0 and Y 0 Perform amplitude compression to obtain a complex spectrum matrix pair (X, Y).

3. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 2, characterized in that: The training process of the discriminant model is: Input the complex matrix Y of the spectrum of noisy speech into the discriminant model D θ , output the estimated value of the complex matrix of the spectrum of the pure speech Constructing reconstruction loss function based on pure speech where L(·) represents the difference between the two; to minimize the loss function For the training target, perform gradient back propagation and update the parameters θ; When the preset training requirements are met, stop training; Select the optimal parameter θ during training * The corresponding discriminant model is used as the target discriminant model.

4. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 3, characterized in that: The process of obtaining the mean-reversal Schrödinger bridge is: For a spectral complex matrix pair (X,Y), assume that the posterior mean of X conditioned on Y is the variable When Y is given, the variable From a conditional posterior distribution It is observed that the target discriminant model is used Approximate the conditional posterior distribution, that is The mean reversal process at a specific time The state sample is represented as sample Obey marginal Gaussian distribution in, represents a uniform distribution on the interval [0,1], Define a Schrödinger bridge process that bridges the noisy speech distribution and the clean speech distribution, and represent the state sample of the Schrödinger bridge process at time t as X t , sample X t Obeying the marginal Gaussian distribution q(X t |X0,X1), and X0=X, X1=Y; at time t, assuming X t by As a conditional representation, the approximate edge Gaussian distribution of the Schrödinger bridge is derived as: in, yes Point estimate of The Schrödinger bridge process that introduces the mean reversal process is called the mean reversal Schrödinger bridge, which replaces the boundary condition X0=X of the Schrödinger bridge at time t with 5. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 4, characterized in that: The mean reversal process is a Schrodinger bridge process that bridges the posterior mean distribution and the pure speech distribution.

6. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 4, characterized in that: At time t, the marginal Gaussian distribution of the mean reversal process is: in, represents the mean of the marginal distribution of the mean reversal process at time t, represents the variance of the marginal distribution of the mean reversal process at time t, α t and σ t is the noise schedule of the mean reversal process, the signal-to-noise ratio 7. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 6, characterized in that: The mean-reversal Schrödinger bridge is described by the following two stochastic differential equations: Where f is the shift coefficient, g is the diffusion coefficient, W and is a standard Wiener process, Ψ t and is the Schrödinger factor, the fraction and represents the forward optimal shift score and the reverse optimal shift score; At time t, the edge Gaussian distribution of the mean-reversed Schrödinger bridge is: in, represents the mean of the edge distribution of the mean-reversed Schrödinger bridge at time t, represents the variance of the mean-reversed Schrödinger bridge edge distribution at time t, and is the noise schedule of the mean-reversal Schrödinger bridge, which is determined by f and g in the above stochastic differential equation, and the signal-to-noise ratio 8. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 7, characterized in that: The training process of the score model is: Input the complex matrix Y of the spectrum of the noisy speech into the target discriminant model Output Random sampling time According to the marginal Gaussian distribution of the mean reversal process at time t Calculate the mean of the distribution and get the point estimate at time t Next, the edge Gaussian distribution of the Schrödinger bridge is inverted according to the mean at time t Sampling to get the sample at time t in is random noise; finally, the time t, sample X t Input score model Output The actual reverse optimal offset score at this time for The estimated inverse optimal shift score is The score model parameters are adjusted based on the difference between the estimated reverse optimal offset score and the actual reverse optimal offset score. The difference between the two is reflected in the difference between X0 and Specifically: define the data prediction loss function To minimize the loss function For the training target, perform gradient back propagation and update parameters When the preset training requirements are met, stop training; Select the optimal parameters during training The corresponding score model is used as the target score model.

9. The single-channel speech enhancement method based on mean-reversal Schrödinger bridge according to claim 8, characterized in that: The reverse generation process is specifically as follows: Set the time t j ,sample Input the target score model and get pass Estimating the inverse optimal offset score Solve inverse stochastic differential equations calculate Let j = j-1. If j>0, repeat the above process. If j = 0, stop the reverse generation. The sample X at time t0 is t0 The complex matrix of the spectrum of the clean speech after the noisy speech enhancement is then subjected to inverse amplitude compression, inverse Fourier transform and inverse normalization to finally obtain the waveform of the clean speech.

10. A device for implementing the single-channel speech enhancement method based on mean-reversed Schrödinger bridge according to any one of claims 1 to 9, characterized in that: include: Construct a sample pair module, which consists of a speech sample pair set consisting of a clean speech sample and a noisy speech sample; A sample processing module performs preprocessing and Fourier transformation on the speech samples in the speech sample pair set to obtain a spectrum complex matrix pair; A discriminant model building module is used to build a discriminant model, estimate the spectral complex matrix of clean speech, and adjust the parameters of the model based on the difference between the estimated spectral complex matrix of clean speech and the spectral complex matrix of real clean speech. When the model meets the preset training requirements, the discriminant model corresponding to the optimal parameters is used as the target discriminant model; A score model construction module is used to construct a score model, estimate the reverse optimal offset score during the reverse generation process, adjust the parameters of the model based on the difference between the estimated reverse optimal offset score and the true reverse optimal offset score, and when the model meets the preset training requirements, use the score model corresponding to the optimal parameters as the target score model; The clean speech generation module, for a given noisy speech, uses the trained target score model to parameterize the inverse optimal offset score of the mean-inverted Schrödinger bridge, performs a reverse generation process, and generates clean speech.

Images