Schrodinger bridge-based diffusion model speech enhancement method and system

Abstract

The invention belongs to the technical field of speech enhancement, and discloses a Schrodinger bridge-based diffusion model speech enhancement method and system, and the method comprises the steps: converting a diffusion process of a diffusion model into a solving process of a stochastic differential equation, and determining the stochastic differential equation according to the theoretical principle of a Schrodinger bridge, and the complex spectrum is directly used as the input of the diffusion model, so that the tedious work of extracting a phase spectrum and an amplitude spectrum from the complex spectrum and the memory overhead caused by inverse transformation are omitted, and meanwhile, the alignment problem between the phase spectrum and the amplitude spectrum is also avoided. According to the speech enhancement method, unique features of time sequence signals are captured through a Transformer module, multi-scale information is fused through a U-Net module, a loss function covering a time domain, a frequency domain and a time-frequency domain is matched, the difference between a prediction sample and a clean sample is gradually reduced, the nonlinear diffusion process from a noisy sample to the clean sample can be directly learned, and the speech enhancement accuracy is improved. Therefore, the structure information of more initial samples is reserved.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech enhancement, and in particular relates to a speech enhancement method and system based on a diffusion model of a Schrödinger bridge. Background Art

[0002] Traditional speech enhancement methods include spectral subtraction and Wiener filtering. Among them, spectral subtraction is to directly subtract the estimated noise power spectrum from the power spectrum of noisy speech. The improved algorithm of related spectral subtraction is more of an improvement on the noise estimation algorithm. However, due to the non-stationarity of noise, speech enhancement methods based on spectral subtraction have the problem of residual noise or even distortion caused by linear power subtraction. The Wiener filtering algorithm is an optimal filtering algorithm based on the minimum mean square error criterion. It filters out noise by designing filters. The improved algorithm of the related Wiener filtering algorithm often optimizes the filter coefficients from multiple angles. However, the speech enhancement method based on the Wiener filtering algorithm relies only on the single minimum mean square error criterion, which is difficult to cope with complex and changeable noise scenes.

[0003] Since traditional speech enhancement methods have weak processing capabilities for nonlinear and non-Gaussian noise, and may introduce distortion while removing noise, they are being replaced by large model methods, such as diffusion models. Diffusion models have been used in generation tasks and have significant advantages. When they are applied to the field of speech enhancement, they also show significant characteristics.

[0004] The diffusion model is divided into a forward diffusion process and a reverse diffusion process. The forward diffusion process is the process of gradually adding Gaussian noise to noisy speech until the data is completely converted to pure Gaussian noise, and the reverse diffusion process is the process of recovering clean speech from the pure Gaussian noise generated by the forward diffusion process. For example, the score-based generative diffusion model speech enhancement method guides the process of gradually recovering clean speech from noise by learning the gradient of data distribution, i.e., the score. It uses a loss function based on score matching to minimize the difference between the predicted score and the true score, and trains a neural network model to estimate the score, thereby realizing the reverse diffusion process.

[0005] Although the score-based generative diffusion model speech enhancement method has achieved good results in the field of speech enhancement, this type of speech enhancement method still has shortcomings. For example, it can only generate data from known distributions such as Gaussian noise, lacks the ability to generate from any distribution to any distribution, and its learning process is also one-way, which leads to slow training speed and limited scope. Summary of the invention

[0006] The purpose of the present invention is to propose a diffusion model speech enhancement method based on Schrödinger bridge, which performs speech enhancement based on the diffusion model, converts the diffusion process of the diffusion model into a solution process of a stochastic differential equation, and determines the stochastic differential equation based on the theoretical principle of the Schrödinger bridge. This method can directly learn the nonlinear diffusion process from noisy samples to clean samples, thereby retaining more structural information of the initial samples.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme: A Schrodinger bridge-based diffusion model speech enhancement method comprises the following steps: Step 1. Convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation; Step 2. Determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, use variance explosion to set the diffusion function and drift function in the stochastic differential equation, and then obtain the representation of the solution of the stochastic differential equation; Step 3. Build a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation; Step 4. Introduce the complex spectrum, amplitude spectrum, phase spectrum, time domain signal and the loss function of the measurement parameters of the time domain signal to train the prediction network model; Step 5. Use the trained prediction network model to output prediction samples for the input noisy speech, and obtain the enhanced speech based on the representation of the solution of the stochastic differential equation.

[0008] In addition, based on the diffusion model speech enhancement method based on Schrödinger bridge, the present invention also proposes a diffusion model speech enhancement system based on Schrödinger bridge adapted thereto, and its technical solution is as follows: A Schrodinger bridge-based diffusion model speech enhancement system, comprising: A conversion module, used to convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation; The stochastic differential equation solving module is used to determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, set the diffusion function and drift function in the stochastic differential equation by using variance explosion, and then obtain the representation of the solution of the stochastic differential equation; The model building module builds a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation; A model training module is used to introduce complex spectrum, amplitude spectrum, phase spectrum, time domain signal and loss function of measurement parameters of time domain signal to train the prediction network model; And a speech enhancement module is used to use the trained prediction network model to output prediction samples for the input noisy speech, and obtain enhanced speech according to the representation of the solution of the stochastic differential equation.

[0009] In addition, based on the above-mentioned diffusion model speech enhancement method based on Schrödinger bridge, the present invention also proposes a computer device, which includes a memory and one or more processors; The memory stores executable codes, and when the processor executes the executable codes, the executable codes are used to implement the steps of the above-mentioned Schrödinger bridge-based diffusion model speech enhancement method.

[0010] In addition, based on the above-mentioned diffusion model speech enhancement method based on Schrödinger bridge, the present invention also proposes a computer-readable storage medium on which a program is stored; when the program is executed by a processor, it is used to implement the steps of the above-mentioned diffusion model speech enhancement method based on Schrödinger bridge.

[0011] The present invention has the following advantages: As described above, the present invention discloses a speech enhancement method based on a diffusion model of Schrödinger bridge. The method constructs a stochastic differential equation based on the Schrödinger bridge, and converts the diffusion process into a solution process of the stochastic differential equation; and directly uses the complex spectrum as the input of the diffusion model, eliminating the tedious work of extracting the phase spectrum and the amplitude spectrum from the complex spectrum and the memory overhead caused by the inverse transformation, and also avoids the alignment problem between the phase spectrum and the amplitude spectrum; the speech enhancement method based on the diffusion model of Schrödinger bridge of the present invention uses the Transformer module to capture the unique characteristics of the time series signal, and then uses the U-Net module to fuse multi-scale information, and matches the loss function covering the time domain, frequency domain and time-frequency domain, and gradually reduces the difference between the predicted sample and the clean sample, so as to achieve better speech denoising and enhancement effect, and the output enhanced speech is superior to the traditional speech enhancement method in terms of objective evaluation indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The flowchart of the diffusion model speech enhancement method based on Schrödinger bridge in an embodiment of the present invention.

[0013] Figure 2 2 is a network structure diagram of the Transformer module in an embodiment of the present invention.

[0014] Figure 3 Schematic diagram of the network structure of the U-Net module in an embodiment of the present invention.

[0015] Figure 4 4 is a network structure diagram of a downsampling block in an embodiment of the present invention.

[0016] Figure 5 4 is a network structure diagram of an upsampling block in an embodiment of the present invention.

[0017] Figure 6 is a network structure diagram of the residual block in an embodiment of the present invention.

[0018] Figure 7 It is a training loss curve diagram of the training process in an embodiment of the present invention.

[0019] Figure 8 It is a verification loss curve diagram of the training process in an embodiment of the present invention.

[0020] Fig. 9 It is a PESQ curve diagram of the training process in an embodiment of the present invention.

[0021] Fig.10 It is a STOI curve diagram of the training process in an embodiment of the present invention.

[0022] Fig.11 It is a schematic diagram of the placeholder of the speech enhancement method of the present invention in the field of speech enhancement in recent years. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 In this embodiment, a diffusion model speech enhancement method based on Schrödinger bridge is proposed. This method performs speech enhancement based on the diffusion model and converts the diffusion process of the diffusion model into the solution process of the stochastic differential equation. Then, the stochastic differential equation is determined according to the theoretical principle of Schrödinger bridge, and the diffusion function and drift function in the stochastic differential equation are set by using variance explosion. Finally, the solution of the stochastic differential equation is obtained. The solution process is a process of sample iteration and feature extraction. The process quantity involved in the diffusion process in the solution representation is the prediction sample. The prediction network model is built through the Transformer module and the U-Net module. , by the prediction network model Output prediction samples.

[0024] like Figure 1 As shown, a diffusion model speech enhancement method based on Schrödinger bridge specifically includes the following steps: Step 1. Perform speech enhancement based on the diffusion model and transform the diffusion process of the diffusion model into the solution process of the stochastic differential equation.

[0025] For the speech enhancement process, the stochastic differential equation that models the forward diffusion process is: (1) in, express The state of the moment, is the drift function, is the diffusion function, is a standard Wiener process.

[0026] The stochastic differential equation modeling the back diffusion process is: (2) in, is the inverse Wiener process, represents the gradient operator, Represents the probability density function The logarithmic gradient of .

[0027] The score function is , the output of the scoring function is the predicted score , prediction score Cannot be used to directly represent prediction samples , but need to go through To represent the prediction sample ,in represents Gaussian noise, Indicates the current time The standard deviation of The representation form cannot directly predict samples, but describes the predicted samples through variance and noise, while the method of the present invention can directly predict samples, which indicates that the diffusion model based on Schrödinger bridge proposed in the present invention is superior to the score-based generation model in the field of speech enhancement.

[0028] Step 2. Determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, use variance explosion to set the diffusion function and drift function in the stochastic differential equation, and then obtain the representation of the solution of the stochastic differential equation.

[0029] The Schrödinger bridge is defined as the path and reference path The Kullback-Leibler divergence between The minimization problem is: (3) Among them, [0,T] represents a time interval, and T is just a generalized symbol representing time.

[0030] The optimal solution to the minimization problem is given by a pair of symmetric forward and backward stochastic differential equations: (4) (5) Formula (4) and Formula (5) are determined based on Formula (1) and Formula (2) according to Formula (3). and is the nonlinear drift term, and denote the optimal forward and optimal reverse drift functions, respectively. and Satisfies the coupled partial differential equations: (6) (7) Where f is the drift function , Represents a trace operation.

[0031] The probability density function of Expressed as ,So and ,in represents the probability density at time t=0, represents the positive drift function at t=0, represents the reverse drift function at time t=0, represents the probability density at time t=T, represents the positive drift function at time t=T, Represents the reverse drift function at time t=T.

[0032] Set two Gaussian boundary conditions and ,in represents the identity matrix, represents the state at time t=0, represents the state at time t=T, represents the variance at time t=0, represents the variance at time t=T, ,So and .

[0033] According to the properties of Gaussian distribution and conditional probability, we can deduce and ,in represents the drift accumulation in [0, t], , Represents the normalization parameter, which is used to associate the intermediate time t with the terminal time T. , represents the drift accumulation in [0,T]; represents the diffusion accumulation in [0, t], , represents the diffusion accumulation in [0,T], .

[0034] Will Substituting into the inverse stochastic differential equation shown in formula (5), we get: (8) After comparative testing, it was found that the solution of the random differential equation has a high delay compared with the solution of the ordinary differential equation, which will cause auditory perception disorders, so the solution of the ordinary differential equation is used. The random differential equation belongs to a random process, which contains random noise terms. Randomness needs to be introduced when generating samples, while the ordinary differential equation belongs to a deterministic process. The ordinary differential equation has no random terms and generates samples by integration. Based on the prerequisite of two Gaussian boundary distributions, the random differential equation can be approximated by the ordinary differential equation.

[0035] Since the direct solution of the stochastic differential equation shown in formula (8) requires the introduction of random terms, the computational complexity is extremely high. and Under this condition, the diffusion term in formula (8) can be approximated by a deterministic integral term, thereby transforming formula (8) into an ordinary differential equation for probability flow.

[0036] The forward process of an ordinary differential equation is: (9) According to Nelson's identity formula, we have , put it into formula (9), and get: (10) Will and Substituting into formula (10), we get: (11) in, The process state, i.e. the process quantity involved in the diffusion process, is represented in the form of a solution to a stochastic differential equation.

[0037] Solving the equation according to formula (11) yields the expression of the solution of the stochastic differential equation: for: (12) in, , , represents the initial parameters. In this embodiment, by setting t=1, , , These three parameters. represents the initial state of each iteration, follow Keep updating and iterating, and Set it as a noisy sample and find the output of the first iteration process , and the current As the initial state of the second iteration . Represents the prediction network model exist The process status of the output at all times; Represents the final state of the forward diffusion process, the final state is the complex spectrum of the noisy samples.

[0038] right The iterative update process corresponds to the reverse diffusion process based on the diffusion model. The equally spaced sampling of It can not only retain the information of noisy speech, but also slowly generate clean speech. , the sampling interval is 0.02, that is, each sample diffusion process requires 50 samples in total.

[0039] In the specific solution process, it is found that variance explosion can provide better sample quality than variance preservation. Therefore, the variance explosion solution is used to set the diffusion function and drift function in the stochastic differential equation. , where c and k are hyperparameters, c and k control the diffusion amplitude and rate respectively. , , = 0, the variance explosion process omits the drift term and drives the variance growth through the diffusion term, so , , .

[0040] Step 3. Build a prediction network model using the Transformer module and the U-Net module , prediction network model Used to output prediction samples , that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation.

[0041] The Transformer module is able to capture the dependencies between time and frequency and can thoroughly learn the properties of signal features in the time domain. Figure 2As shown, the Transformer module designed by the present invention includes a normalization layer, a multi-head self-attention mechanism MHSA, a fully connected network FFN, and a regularized Dropout layer. Among them, the normalization layer is used to maintain the stable distribution of input data in each network layer, alleviate the common gradient disappearance or explosion problems of deep neural networks during training, so that the network can converge more smoothly and quickly. The Dropout layer randomly discards neurons with a certain probability to prevent over-reliance on specific neurons, forcing the network to learn more robust and more general feature representations, enhance the generalization ability of the network, and avoid overfitting. MHSA can make the model understand and represent input data more carefully and accurately, and improve the overall feature extraction ability of the network. FFN performs deep nonlinear transformation on the extracted features, which helps to mine deeper feature information.

[0042] It is worth mentioning that at the input end of MHSA, a unique method is used to process the complex spectrum, which is different from the traditional speech enhancement method that extracts phase information and amplitude information from the complex spectrum. In order to avoid the memory consumption caused by the extraction of phase information and amplitude information and the problem of difficulty in aligning the two, the present invention adopts the following method: First, the input noisy speech, i.e. the noisy sample, is normalized and short-time Fourier transformed (STFT) to obtain the complex spectrum. If this is the first iteration, you also need to Initialize it so that = . The output of the iteration Complex spectrum of noisy samples Integrate according to the preset dimension to obtain a mixed complex spectrum ,in Represents the time domain frame number, represents the Fourier length, Represents the batch size. In order to match the input of the MHSA network, the mixed complex spectrum is reorganized by reorganizing the multidimensional tensor on the preset dimension. Reshape The dimensionality of the multidimensional tensor is changed only to adapt to the Transformer network structure. It is a common processing method to reorganize the multidimensional tensor according to a certain dimension. Finally, after passing through the Transformer module, it is restored to , will soon After entering the Transformer module, the output is .

[0043] The U-Net module is composed of a convolutional autoencoder with skip connections. It is very effective in generation and segmentation tasks. The structure is similar to the U-type and is divided into a downsampling module and an upsampling module. The downsampling module is used to extract features, reduce resolution, and increase the number of feature channels. The upsampling module restores the resolution of the extracted features and accurately reconstructs them by combining them with the features from the corresponding encoder layer.

[0044] like Figures 3 to 6 As shown, the network depth of the U-Net module designed by the present invention is 7. The U-Net module includes seven down-sampling modules and seven up-sampling modules.

[0045] Each downsampling module includes two branches, the first branch of the downsampling module includes two residual blocks and a downsampling block, the second branch of the downsampling module includes a downsampling operation module, and the two residual blocks in the downsampling module are respectively defined as a first residual block and a second residual block.

[0046] Each upsampling module includes two branches, the first branch of the upsampling module includes three residual blocks and an upsampling block, the second branch of the upsampling module includes an upsampling operation module, and the three residual blocks in the upsampling module are respectively defined as a third residual block, a fourth residual block and a fifth residual block.

[0047] The signal processing flow in the U-Net module is as follows: The output of the Transformer module As the input of the U-Net module, input the First, the number of channels is increased to 128 through the convolutional network, and we get .

[0048] Will As the input of the first branch and the second branch of the first downsampling module, a set of eigenvalues ​​are output after passing through the first residual block, the second residual block and a downsampling block in the first branch of the downsampling module in sequence, and a set of eigenvalues ​​are output after performing a downsampling operation in the second branch of the downsampling module. The outputs of the first branch and the second branch of the downsampling module are linearly fused to obtain the output of the first downsampling module, and the output of the current downsampling module is used as the input of the next downsampling module, and is processed in sequence through all the downsampling modules in the U-Net module.

[0049] The output of the first downsampling module passes through a residual block and outputs a set of eigenvalues, which are used as the input of the first branch and the second branch of the first upsampling module; the eigenvalues ​​in the first branch of the input upsampling module are concatenated with the output of the downsampling block in the downsampling module of the corresponding level through a jump connection and input into the third residual block; the output of the third residual block is concatenated with the output of the second residual block in the downsampling module of the corresponding level through a jump connection and input into the fourth residual block; the output of the fourth residual block is concatenated with the output of the first residual block in the downsampling module of the corresponding level through a jump connection and input into the upsampling block, and the output of the upsampling block is used as the output of the first branch of the upsampling module; the eigenvalues ​​in the second branch of the input upsampling module are upsampled and used as the output of the second branch of the upsampling module; the outputs of the first branch and the second branch of the upsampling module are linearly fused to obtain the output of the current upsampling module, and used as the input of the next upsampling module, processed in turn by all the upsampling modules in the U-Net module, and the output of the last upsampling module is used as the prediction network model Output .

[0050] The difference in the number of residual blocks in the U-Net module is due to the fact that during the downsampling process, there are three features at each resolution. These three features are obtained through the downsampling operation of the previous downsampling module and two identical convolution steps, namely, the three sets of feature values ​​output by the first residual block, the second residual block, and the downsampling block, covering adjacent feature information, and there is a strong correlation between them. Therefore, in the upsampling module, these three features need to be collected and included in the feature recovery. Therefore, the upsampling module needs to accumulate residuals from the downsampling module of the same resolution, which is more complicated than the downsampling module. In addition, except for the upsampling and downsampling operations, the other structures of the upsampling and downsampling blocks are the same.

[0051] Step 4. Consider the complex spectrum, phase spectrum, amplitude spectrum in the time domain, frequency domain, time-frequency domain, time domain signal, and measurement parameters of the time domain signal, and introduce the loss function to predict the network model. Conduct training.

[0052] The present invention introduces five loss functions, covering the complex spectrum, phase spectrum, amplitude spectrum in the time domain, frequency domain, time-frequency domain, and time domain signal and the measurement parameters of the time domain signal. For the complex spectrum loss, the present invention adopts L2 loss : (13) in, For predicting network models The output of is the predicted sample, x represents the complex spectrum of clean speech, that is, the clean sample, and the loss function is to find the difference between the two complex spectra. It represents the expected value. The loss function generally sums the difference between two vectors and then takes the average. This process is generally represented by E.

[0053] Since the prediction network model Only the complex spectrum is processed, and the phase spectrum and amplitude spectrum are not processed separately. Therefore, in order to balance the integrity of the feature information, the present invention introduces the phase spectrum loss function And the amplitude spectrum loss function , the amplitude spectrum loss function is: (14) in, and They are and Amplitude information, Indicates expected value.

[0054] Due to the periodicity of trigonometric functions, the phase is usually in the interval or , when calculating the signal phase information, this limit may be exceeded, resulting in phase wrapping. The present invention uses the phase unwrapping function First, control the phase information to interval, and then calculate the phase loss, The expression is as follows: (15) When calculating the phase loss, the present invention introduces three types of phase loss, namely instantaneous phase loss , instantaneous angular frequency loss , Group Delay Loss Because the instantaneous phase reflects the actual phase of the signal at each instant, the angular frequency determines the basic rate at which the signal phase changes over time, and the group delay reflects the time delay difference of different frequency components during the transmission process, only by comprehensively considering these three factors can the phase change of the signal be fully and accurately described.

[0055] (16) (17) (18) in, and They are and The phase information of and denote the differential operators along the time axis and frequency axis respectively, , , represents the expected value, the total phase loss It is three components , , sum: (19) Prediction Network Model The ultimate goal is to minimize the difference between the predicted sample and the clean sample, including the time domain range. The complex spectrum must first be converted into a time domain signal through the inverse short-time Fourier transform ISTFT, so the time domain loss function : (20) in, and They are and The time domain signal, Indicates expected value.

[0056] The metric loss is based on the objective speech quality assessment PESQ score, which has been proven to be very effective in the field of speech enhancement. The present invention uses the existing library function: (twenty one) in, The loss function representing the metric parameters of the time domain signal, Indicates the speech quality judgment index PESQ.

[0057] The overall loss function L is the sum of five loss functions, as follows: (twenty two) in, , , , Represents the weight coefficient. In this embodiment =0.1, =0.0001, =0.01, =0.01. The speech signal features can be expressed as time-frequency domain features and time-domain features. The loss functions of these two parts account for the largest proportion. However, because the input and output of the model in the method of the present invention are based on the complex spectrum in the time-frequency domain, this embodiment sets the complex spectrum loss as the benchmark, and the complex spectrum loss function The weight of is 1. Time domain loss function Belongs to the auxiliary constraint, and its weight The phase loss and amplitude loss of speech signals are two quantities obtained simultaneously based on the complex spectrum. They have the same weight. Compared with the global features such as time-frequency domain features and time domain features, they are detail features. Therefore, the phase spectrum loss function And the amplitude spectrum loss function The weight of the feature is also smaller than that of the global feature. and With the time domain loss function Weight As for the metric loss, a single metric loss only plays a leading role and cannot occupy a large weight. A large weight will only guide the model to train for the purpose of improving indicators, rather than for the perfect approximation of global features. Therefore, the loss function of the metric parameter of the time domain signal is Weight The lowest proportion.

[0058] Step 5. Use the trained prediction network model to output prediction samples for the input noisy speech, and obtain the enhanced speech based on the representation of the solution of the stochastic differential equation.

[0059] In this embodiment, step 5 is specifically as follows: Step 5.1. Perform normalization and STFT on the input noisy speech to obtain the complex spectrum Y of the noisy speech.

[0060] Step 5.2. Initialize to , initialize the time to t=T, and obtain the initial , , , ; Set the number of steps n and step length N in the interval [0,T] t , and satisfy nN t =T.

[0061] In this embodiment, T is set to 1, the number of steps is 50, and the step length is 0.02. Then the time is initialized to t=1, and the number of steps is set to 50 and the step length is set to 0.02 in the interval [0,1], and the set of t is [1,0.98,…,0.02,0].

[0062] Step 5.3. Update the value of t to t=tN t , and find the current and .

[0063] The parameters , Y, t are input into the trained prediction network model In the prediction network model, the prediction sample is output .

[0064] According to the expression of the solution of the differential equation shown in formula (12), the solution of the differential equation in the iterative process is obtained ,in is the complex spectrum Y of the noisy speech.

[0065] Step 5.4. Determine whether the current time t is 0; if so, set the current time As the final state of the reverse diffusion process, go to step 5.5; otherwise, the current moment Set as the initial state for the next iteration, , the current moment and Assign value to next iteration and ,Right now , , and go to step 5.3.

[0066] Step 5.5. Final state Perform inverse STFT and denormalization processing to obtain enhanced speech.

[0067] The purpose of the model is to transform noisy samples into clean samples, so the noisy samples are known. In this embodiment, the reverse diffusion process is to update the noisy samples 50 times, each time updating the weights of the current round of prediction samples, noisy samples, and the previous round of samples, and slowly restore them to clean samples. and is constantly changing, and at the end of this round Assigned to ,Will Assigned to , to facilitate the next round of use, this is and Update, and is fixed. The diffusion process is During the iterative update process, It's from the last round. , after completing the iteration, we finally get is the final state of the reverse diffusion process. Formula (12) is the solution of the reverse diffusion process. In formula (12) It is the final state of the forward diffusion process.

[0068] The speech enhancement algorithm of the present invention is based on the Schrödinger bridge optimization path principle, that is, finding the optimal path between the initial data distribution and the target data distribution, minimizing the Kullback-Leibler divergence of the two distributions, and constructing the optimal transmission path from the initial probability distribution to the target probability distribution. Based on this principle, the diffusion process is converted into a solution process of a stochastic differential equation, aiming to approximate noisy samples to clean samples through probability distribution. Unlike the diffusion model that first learns the forward diffusion process from noisy samples to Gaussian distribution, the speech enhancement algorithm proposed in the present invention directly learns the nonlinear diffusion process from noisy samples to clean samples, thereby retaining more structural information of the initial samples.

[0069] The speech enhancement method of the present invention utilizes the Transformer module to capture the unique characteristics of the time series signal, and pre-processes the complex spectrum by introducing the Transformer network. The complex spectrum can be directly used as the input of the diffusion model, eliminating the tedious work of extracting the phase spectrum and amplitude spectrum from the complex spectrum and the memory overhead caused by the inverse transformation, while also avoiding the alignment problem between the phase spectrum and the amplitude spectrum. The U-Net module is then used to fuse multi-scale information, and a 7-layer U-Net network is introduced, and a convolutional network is formed with the Transformer network for predicting data extraction. The present invention also takes into account the contribution of phase and amplitude to speech enhancement, adds phase loss and amplitude loss to the loss function, sets ideal weights, and matches 5 loss functions covering the time domain, frequency domain, and time-frequency domain to gradually reduce the difference between the predicted sample and the clean sample.

[0070] The output speech obtained by the speech enhancement method of the present invention achieved evaluation results of 3.79, 4.72, 3.73, 4.36, and 0.95 in five objective evaluation indicators: PESQ speech quality perception evaluation, CSIG speech distortion, CBAK background noise interference, COVL overall speech quality, and STOI short-time objective intelligibility, respectively. Compared with the speech enhancement models in recent years, the best performance was achieved.

[0071] Figure 7 The entire training process in the embodiment of the present invention is shown through the loss curve of the training set. Figure 8 The entire training process in the embodiment of the present invention is shown through the loss curve of the validation set. Figure 7 and Figure 8 It is shown that during the entire training process, the loss function of the training set and the loss function of the validation set converge quickly to stability, which on the one hand reflects that the training of the network in the present invention is effective, and on the other hand reflects that the training of the network in the present invention is convergent. Fig. 9 and Fig.10 The effectiveness of the training process is verified by testing the PESQ and STOI indicators during the training process. Fig. 9and Fig.10 It shows that the speech quality indicators PESQ and STOI continue to increase throughout the training process. Fig.11 The mainstream speech enhancement models in recent years are listed, where the horizontal axis is the year and the vertical axis is the PESQ score. Through comparison, it can be seen that the average PESQ score of the speech enhancement model in each year is increasing year by year, indicating the development trend of speech enhancement. The PESQ score of the enhanced speech output by the speech enhancement method of the present invention is the highest at present, which reflects the advanced nature of the speech enhancement method of the present invention.

[0072] Example 2 This embodiment 2 describes a diffusion model speech enhancement system based on Schrödinger bridge, which is based on the same inventive concept as the diffusion model speech enhancement method based on Schrödinger bridge in embodiment 1.

[0073] Specifically, the diffusion model speech enhancement system based on Schrödinger bridge includes the following modules: The conversion module is used to convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation.

[0074] The stochastic differential equation solving module is used to determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, use the variance explosion to set the diffusion function and drift function in the stochastic differential equation, and then obtain the representation of the solution of the stochastic differential equation.

[0075] The model building module builds a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation.

[0076] The model training module is used to introduce the complex spectrum, amplitude spectrum, phase spectrum, time domain signal and the loss function of the measurement parameters of the time domain signal to train the prediction network model.

[0077] And a speech enhancement module is used to use the trained prediction network model to output prediction samples for the input noisy speech, and obtain enhanced speech according to the representation of the solution of the stochastic differential equation.

[0078] It should be noted that, in the diffusion model speech enhancement system based on Schrödinger bridge, the implementation process of the functions and effects of each functional module is specifically described in the implementation process of the corresponding steps in the method in Example 1, and will not be repeated here.

[0079] Example 3 This embodiment 3 describes a computer device, which includes a memory and one or more processors.

[0080] An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the steps of the diffusion model speech enhancement method based on Schrödinger bridge in the above embodiment 1.

[0081] In this embodiment, the computer device is any device or apparatus with data processing capability, which will not be described in detail here.

[0082] Example 4 This embodiment 4 describes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the Schrödinger bridge-based diffusion model speech enhancement method.

[0083] The computer-readable storage medium may be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc., equipped on the device.

[0084] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A Schrodinger bridge-based diffusion model speech enhancement method, characterized in that: The steps include: Step 1. Convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation; Step 2. Determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, use variance explosion to set the diffusion function and drift function in the stochastic differential equation, and then obtain the representation of the solution of the stochastic differential equation; Step 3. Build a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation; Step 4. Introduce the complex spectrum, amplitude spectrum, phase spectrum, time domain signal and the loss function of the measurement parameters of the time domain signal to train the prediction network model; Step 5. Use the trained prediction network model to output prediction samples for the input noisy speech, and obtain the enhanced speech based on the representation of the solution of the stochastic differential equation.

2. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 1, characterized in that: The step 1 is specifically as follows: For the speech enhancement process, the stochastic differential equation modeling the forward diffusion process is: ； in, express The state of the moment, is the drift function, is the diffusion function, is the standard Wiener process; The stochastic differential equation modeling the back diffusion process is: ； in, is the inverse Wiener process, represents the gradient operator, Represents the probability density function The logarithmic gradient of .

3. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 2, characterized in that: The step 2 is specifically as follows: The Schrödinger bridge is defined as the path and reference path The Kullback-Leibler divergence between The minimization problem is: ； Wherein, T represents the time symbol; The solution to the minimization problem is given by a pair of symmetric forward and backward stochastic differential equations: ； ； in, and is the nonlinear drift term, and denote the forward and reverse drift functions, respectively; Get a representation of the solution to the stochastic differential equation for: ； in, , , represents the initial parameters, represents the drift accumulation in [0, t], , represents the drift accumulation in [0,T]; represents the diffusion accumulation in [0, t], , represents the diffusion accumulation in [0,T], , represents the initial state of each iteration, represents the state at time t=T, Process quantities involving diffusion processes in the representation of the process state, i.e., the solution of the stochastic differential equation; Using the variance explosion solution, set , where c and k are hyperparameters, c and k control the diffusion amplitude and rate respectively. =0, so , , .

4. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 3, characterized in that: In step 3, a preprocessing operation is performed before inputting the Transformer module, and the noisy speech, i.e., the noisy sample, is normalized and short-time Fourier transformed (STFT) to obtain a complex spectrum. , in the first iteration Initialize to ; Will Complex spectrum of noisy samples Integrate to get the mixed complex spectrum ,in Represents the time domain frame number, represents the Fourier length, Represents the batch size; The mixed complex spectrum is reorganized by reorganizing the multidimensional tensor on the preset dimension. Reshape ; The Transformer module includes a normalization layer, a multi-head self-attention mechanism MHSA, a fully connected network FFN, and a regularized Dropout layer. After entering the Transformer module, the output is .

5. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 4, characterized in that: In step 3, the U-Net module includes seven down-sampling modules and seven up-sampling modules; Each downsampling module includes two branches, the first branch of the downsampling module includes two residual blocks and a downsampling block, and the second branch of the downsampling module includes a downsampling operation module; Each upsampling module includes two branches, the first branch of the upsampling module includes three residual blocks and an upsampling block, and the second branch of the upsampling module includes an upsampling operation module.

6. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 5, characterized in that: The step 4 is specifically as follows: For the complex spectrum loss, L2 loss is used : ； in, For predicting network models The output of is the predicted sample, x represents the complex spectrum of clean speech, i.e., the clean sample. Indicates expected value; Introducing the amplitude spectrum loss function : ； in, and They are and Amplitude information, Indicates expected value; Using the phase unwrapping function The phase information is controlled interval, The expression is: ； Calculate the instantaneous phase loss separately , instantaneous angular frequency loss , Group Delay Loss : ； ； ； in, and They are and The phase information of and denote the differential operators along the time axis and frequency axis respectively, , , Indicates expected value; The total phase spectrum loss function It is three components , , sum: ； Time Domain Loss Function for: ； in, and They are and The time domain signal, Indicates expected value; The loss metric is based on the objective speech quality assessment PESQ score: ； in, The loss function representing the metric parameters of the time domain signal, Indicates the speech quality judgment index PESQ; The overall loss function L is obtained as: ； in, , , , Represents the weight coefficient.

7. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 6, characterized in that: The step 5 is specifically as follows: Step 5.

1. Perform normalization and STFT on the input noisy speech to obtain the complex spectrum Y of the noisy speech; Step 5.

2. Initialize to ,Will Initialize to , initialize the time to t=T, and obtain the initial , , , ; Set the number of steps n and the step length N in the interval [0,T] t , and satisfy nN t =T; Step 5.

3. Update the value of t to t=tN t , and find the current and ; The parameters , Y, t are input into the trained prediction network model In the prediction network model, the prediction sample is output ; According to the representation of the solution of the stochastic differential equation, The complex spectrum Y of the noisy speech is obtained, and the solution of the differential equation in the iterative process is used to update the current moment ; Step 5.

4. Determine whether the current time t is 0; if so, set the current time As the final state of the reverse diffusion process, go to step 5.5; otherwise, the current moment Set as the initial state for the next iteration, , and the current moment and Assign value to next iteration and ,Right now , , then go to step 5.3; Step 5.

5. Final state of the back diffusion process Perform inverse STFT and denormalization processing to obtain enhanced speech.

8. A diffusion model speech enhancement system based on Schrödinger bridge, characterized in that: include: A conversion module, used to convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation; The stochastic differential equation solving module is used to determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, set the diffusion function and drift function in the stochastic differential equation by using variance explosion, and then obtain the representation of the solution of the stochastic differential equation; The model building module builds a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation; A model training module is used to introduce complex spectrum, amplitude spectrum, phase spectrum, time domain signal and loss function of measurement parameters of time domain signal to train the prediction network model; And a speech enhancement module is used to use the trained prediction network model to output prediction samples for the input noisy speech, and obtain enhanced speech according to the representation of the solution of the stochastic differential equation.

9. A computer device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the steps of the Schrödinger bridge-based diffusion model speech enhancement method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the Schrödinger bridge-based diffusion model speech enhancement method as described in any one of claims 1 to 7 are implemented.

Images