A Speech Enhancement Method and System Based on Schrödinger Bridge Diffusion Model

Abstract

The present invention belongs to the technical field of speech enhancement, and discloses a speech enhancement method and system based on the Schrödinger bridge diffusion model. This method transforms the diffusion process of the diffusion model into a solution process of a stochastic differential equation, determines the stochastic differential equation according to the theoretical principle of the Schrödinger bridge, and directly uses the complex spectrum as the input of the diffusion model, eliminating the cumbersome work of extracting the phase spectrum and amplitude spectrum from the complex spectrum and the memory overhead caused by the inverse transformation. At the same time, the alignment problem between the phase spectrum and the amplitude spectrum is also avoided. The speech enhancement method of the present invention captures the unique features of the time series signal through the Transformer module, then uses the U-Net module to fuse multi-scale information, and combines loss functions covering the time domain, frequency domain, and time-frequency domain to gradually reduce the difference between the predicted sample and the clean sample, and can directly learn the non-linear diffusion process from the noisy sample to the clean sample, thereby retaining more structural information of the initial sample.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech enhancement, and particularly relates to a speech enhancement method and system based on a Schrödinger bridge diffusion model. Background Art

[0002] Traditional speech enhancement methods include spectral subtraction and Wiener filtering algorithms. Among them, spectral subtraction directly subtracts the estimated noise power spectrum from the power spectrum of noisy speech. More improved algorithms related to spectral subtraction are mainly improvements to the noise estimation algorithm. However, due to the non-stationarity of noise, the speech enhancement method based on spectral subtraction has problems such as residual noise and even distortion caused by linear subtraction of power. The Wiener filtering algorithm is an optimal filtering algorithm based on the minimum mean square error criterion. It filters out noise by designing a filter. Improved algorithms related to the Wiener filtering algorithm often optimize the filter coefficients from multiple angles. However, the speech enhancement method based on the Wiener filtering algorithm relies only on a single minimum mean square error criterion and is difficult to handle complex and variable noise scenarios.

[0003] Since traditional speech enhancement methods have weak processing capabilities for non-linear and non-Gaussian noise and may introduce distortion while removing noise, they are being replaced by large model methods, such as diffusion models. Diffusion models have been applied to generation tasks and have shown significant advantages. When applied to the field of speech enhancement, they also exhibit remarkable features.

[0004] The diffusion model is divided into a forward diffusion process and a reverse diffusion process. The forward diffusion process is a process of gradually adding Gaussian noise to noisy speech until the data is completely converted into pure Gaussian noise. The reverse diffusion process is a process of recovering clean speech from the pure Gaussian noise generated by the forward diffusion process. For example, the speech enhancement method based on the score-based generative diffusion model learns the gradient of the data distribution, i.e., the score, to guide the process of gradually recovering clean speech from noise. It is paired with a loss function based on score matching to minimize the difference between the predicted score and the true score, and trains a neural network model to estimate the score, thereby realizing the reverse diffusion process.

[0005] Although the speech enhancement method based on the score-based generative diffusion model has good results in the field of speech enhancement, such speech enhancement methods still have deficiencies. For example, they can only start generating data from known distributions such as Gaussian noise, lack the generative ability from any distribution to any distribution, and their learning process is also one-way, resulting in slow training speed and limited scope. Summary of the Invention

[0006] The object of the present invention is to propose a diffusion model-based speech enhancement method based on the Schrödinger bridge, which performs speech enhancement based on the diffusion model, transforms the diffusion process of the diffusion model into a solution process of a stochastic differential equation, and determines the stochastic differential equation according to the theoretical principle of the Schrödinger bridge. This method can directly learn the non-linear diffusion process from noisy samples to clean samples, thereby retaining more structural information of the initial samples.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A diffusion model-based speech enhancement method based on the Schrödinger bridge, comprising the following steps:

[0009] Step 1. Transform the diffusion process of the diffusion model into a solution process of a stochastic differential equation;

[0010] Step 2. Determine the stochastic differential equation according to the theoretical principle of the Schrödinger bridge, and use variance explosion to set the diffusion function and drift function in the stochastic differential equation, so as to obtain the representation form of the solution of the stochastic differential equation;

[0011] Step 3. Build a prediction network model through a Transformer module and a U-Net module. The prediction network model is used to output a prediction sample, that is, the process quantity involved in the diffusion process in the representation form of the solution of the stochastic differential equation;

[0012] Step 4. Introduce a loss function of complex spectrum, amplitude spectrum, phase spectrum, time-domain signal and measurement parameters of the time-domain signal to train the prediction network model;

[0013] Step 5. Use the trained prediction network model to output a prediction sample for the input noisy speech, and obtain the enhanced speech according to the representation form of the solution of the stochastic differential equation.

[0014] In addition, on the basis of the diffusion model-based speech enhancement method based on the Schrödinger bridge, the present invention also proposes a corresponding diffusion model-based speech enhancement system based on the Schrödinger bridge. The technical solution is as follows:

[0015] A diffusion model-based speech enhancement system based on the Schrödinger bridge, comprising:

[0016] A transformation module for transforming the diffusion process of the diffusion model into a solution process of a stochastic differential equation;

[0017] A stochastic differential equation solving module for determining the stochastic differential equation according to the theoretical principle of the Schrödinger bridge, and using variance explosion to set the diffusion function and drift function in the stochastic differential equation, so as to obtain the representation form of the solution of the stochastic differential equation;

[0018] The model building module builds a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation;

[0019] A model training module is used to introduce complex spectrum, amplitude spectrum, phase spectrum, time domain signal and loss function of measurement parameters of time domain signal to train the prediction network model;

[0020] And a speech enhancement module is used to use the trained prediction network model to output prediction samples for the input noisy speech, and obtain enhanced speech according to the representation of the solution of the stochastic differential equation.

[0021] In addition, based on the above-mentioned diffusion model speech enhancement method based on Schrödinger bridge, the present invention also proposes a computer device, which includes a memory and one or more processors;

[0022] The memory stores executable codes, and when the processor executes the executable codes, the executable codes are used to implement the steps of the above-mentioned Schrödinger bridge-based diffusion model speech enhancement method.

[0023] In addition, based on the above-mentioned diffusion model speech enhancement method based on Schrödinger bridge, the present invention also proposes a computer-readable storage medium on which a program is stored; when the program is executed by a processor, it is used to implement the steps of the above-mentioned diffusion model speech enhancement method based on Schrödinger bridge.

[0024] The present invention has the following advantages:

[0025] As described above, the present invention discloses a speech enhancement method based on a diffusion model of Schrödinger bridge. The method constructs a stochastic differential equation based on the Schrödinger bridge, and converts the diffusion process into a solution process of the stochastic differential equation; and directly uses the complex spectrum as the input of the diffusion model, eliminating the tedious work of extracting the phase spectrum and the amplitude spectrum from the complex spectrum and the memory overhead caused by the inverse transformation, and also avoids the alignment problem between the phase spectrum and the amplitude spectrum; the speech enhancement method based on the diffusion model of Schrödinger bridge of the present invention uses the Transformer module to capture the unique characteristics of the time series signal, and then uses the U-Net module to fuse multi-scale information, and matches the loss function covering the time domain, frequency domain and time-frequency domain, and gradually reduces the difference between the predicted sample and the clean sample, so as to achieve better speech denoising and enhancement effect, and the output enhanced speech is superior to the traditional speech enhancement method in terms of objective evaluation indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The flowchart of the diffusion model speech enhancement method based on Schrödinger bridge in an embodiment of the present invention.

[0027] Figure 2 This is the network structure diagram of the Transformer module in the embodiment of the present invention.

[0028] Figure 3 This is the network structure diagram of the U-Net module in the embodiment of the present invention.

[0029] Figure 4 This is the network structure diagram of the downsampling block in the embodiment of the present invention.

[0030] Figure 5 This is the network structure diagram of the upsampling block in the embodiment of the present invention.

[0031] Figure 6 This is the network structure diagram of the residual block in the embodiment of the present invention.

[0032] Figure 7 This is the training loss curve graph of the training process in the embodiment of the present invention.

[0033] Figure 8 This is the validation loss curve graph of the training process in the embodiment of the present invention.

[0034] Figure 9 This is the PESQ curve graph of the training process in the embodiment of the present invention.

[0035] Figure 10 This is the STOI curve graph of the training process in the embodiment of the present invention.

[0036] Figure 11 This is the occupancy schematic diagram of the voice enhancement method of the present invention in the field of voice enhancement in recent years. Detailed implementation manners

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners:

[0038] Embodiment 1

[0039] In this embodiment, a voice enhancement method based on the Schrödinger bridge diffusion model is proposed. This method performs voice enhancement based on the diffusion model, transforms the diffusion process of the diffusion model into a solution process of a stochastic differential equation, then determines the stochastic differential equation according to the theoretical principle of the Schrödinger bridge, sets the diffusion function and drift function in the stochastic differential equation using variance explosion, and finally obtains the representation form of the solution of the stochastic differential equation. The solution process is a process of sample iteration and feature extraction, and the process quantity involved in the diffusion process in the representation form of the solution is the predicted sample. A prediction network model is built through the Transformer module and the U-Net module , and the predicted sample is output by the prediction network model .

[0040] AsFigure 1 As shown in Figure 1 , a voice enhancement method based on the Schrödinger bridge diffusion model specifically includes the following steps:

[0041] Step 1. Perform voice enhancement based on the diffusion model and transform the diffusion process of the diffusion model into the solution process of a stochastic differential equation.

[0042] For the voice enhancement process, model the stochastic differential equation of the forward diffusion process:

[0043] (1)

[0044] where represents the state at time is the drift function, is the diffusion function, is the standard Wiener process.

[0045] Model the stochastic differential equation of the reverse diffusion process as:

[0046] (2)

[0047] where is the inverse Wiener process, represents the gradient operator, represents the probability density function of the logarithmic gradient.

[0048] The score function is , and the output of the score function is the predicted score . The predicted score cannot be used to directly represent the predicted sample , but instead needs to be represented by to represent the predicted sample , where represents Gaussian noise, represents the standard deviation at the current time . This representation form cannot directly predict the sample, but describes the predicted sample through variance and noise. However, the method of the present invention can directly predict the sample, indicating that the diffusion model based on the Schrödinger bridge proposed by the present invention is superior to the score-based generative model in the field of voice enhancement.

[0049] Step 2. Determine the stochastic differential equation according to the theoretical principle of the Schrödinger bridge, and use variance explosion to set the diffusion function and drift function in the stochastic differential equation, and then obtain the representation form of the solution of the stochastic differential equation.

[0050] The Schrödinger bridge is defined as the path and the reference path The Kullback-Leibler divergence between them, i.e., the relative entropy minimization problem:

[0051] (3)

[0052] where [0,T] represents a time interval and T is just a symbol generally representing time.

[0053] The optimal solution to the minimization problem is given by a pair of symmetric forward and backward stochastic differential equations:

[0054] (4)

[0055] (5)

[0056] Equations (4) and (5) are determined based on Equations (1) and (2) according to Equation (3). Among them, and are non-linear drift terms, and represent the optimal forward and optimal backward drift functions respectively, and satisfy the coupled partial differential equations:

[0057] (6)

[0058] (7)

[0059] where f is the drift function , represents the trace operation.

[0060] The probability density function of is expressed as , then and , where represents the probability density at t = 0, represents the forward drift function at t = 0, represents the backward drift function at t = 0, represents the probability density at t = T, represents the forward drift function at t = T, represents the backward drift function at t = T.

[0061] Set two Gaussian boundary conditions and , where denotes the identity matrix, denotes the state at time t = 0, denotes the state at time t = T, denotes the variance at time t = 0, denotes the variance at time t = T, , then and .

[0062] Based on the properties of the Gaussian distribution and conditional probability, it is deduced that and , where denotes the cumulative drift within [0, t], , denotes the normalization parameter, which is used to associate the intermediate time t and the terminal time T, , denotes the cumulative drift within [0, T]; denotes the cumulative diffusion within [0, t], , denotes the cumulative diffusion within [0, T], .

[0063] Substitute into the backward stochastic differential equation shown in formula (5), and we get:

[0064] (8)

[0065] Through comparative tests, it is found that the solution of the stochastic differential equation has a high delay compared with the solution of the ordinary differential equation, which will cause auditory perception disorders. Therefore, the solution of the ordinary differential equation is adopted. The stochastic differential equation belongs to the stochastic process, which contains a stochastic noise term and needs to introduce randomness when generating samples, while the ordinary differential equation belongs to the deterministic process. The ordinary differential equation has no random term and generates samples through integration. Based on the prerequisite of two Gaussian boundary distributions, the stochastic differential equation can be approximated by the ordinary differential equation.

[0066] Since directly solving the stochastic differential equation shown in formula (8) requires introducing a stochastic term, the computational complexity is extremely high. At the same time, under the Gaussian boundary conditions and , the diffusion term in formula (8) can be approximated by a deterministic integral term, thus transforming formula (8) into an ordinary differential equation of the probability current.

[0067] The forward process of the ordinary differential equation is:

[0068] (9)

[0069] According to Nelson's identity formula, that is, Nelson's identity, we have , substituting it into formula (9), we get:

[0070] (10)

[0071] Substitute and into formula (10), we get:

[0072] (11)

[0073] Among them, represents the process state, that is, the process quantity involved in the diffusion process in the representation form of the solution of the stochastic differential equation.

[0074] Solve the equation according to formula (11) to obtain the representation form of the solution of the stochastic differential equation as:

[0075] (12)

[0076] Among them, , , represent the initial parameters. In this embodiment, by setting t = 1, we obtain , , these three parameters. represents the initial state of each iteration, follows and is continuously updated iteratively. Set in the first iteration process as the noisy sample, and obtain the output of the first iteration process, and use the current as the initial state of the second iteration. represents the process state output by the prediction network model at time; represents the final state of the forward diffusion process, and the final state is the complex spectrum of the noisy sample.

[0077] The process of iterative update of corresponds to the reverse diffusion process based on the diffusion model. Through equidistant sampling, it can not only retain the information of the noisy speech but also slowly generate clean speech. In this embodiment , the sampling interval is 0.02, that is, a total of 50 samplings are required for each sample diffusion process.

[0078] In the specific solution process, it is found that variance explosion can provide better sample quality than variance preservation. Therefore, the diffusion function and drift function in the stochastic differential equation are set by using the variance explosion solution method. Set , where c and k are hyperparameters, and c and k control the diffusion amplitude and rate respectively. In this embodiment , , =0. In the variance explosion process, the drift term is omitted, and the variance growth is driven by the diffusion term. Therefore , , .

[0079] Step 3. Build a prediction network model through the Transformer module and the U-Net module , the prediction network model is used to output the predicted sample , that is, the process quantity involved in the diffusion process in the representation form of the solution of the stochastic differential equation.

[0080] The Transformer module can capture the dependencies between time and frequency and can thoroughly learn the characteristics of signal features in the time domain. As Figure 2 shown, the Transformer module designed in the present invention includes a normalization layer, a multi-head self-attention mechanism MHSA, a fully connected network FFN, and a regularization Dropout layer. Among them, the normalization layer is used to maintain the stable distribution of the input data in each network layer, alleviate the common gradient disappearance or explosion problems in the training process of deep neural networks, so that the network can converge more smoothly and quickly. The Dropout layer randomly discards neurons with a certain probability to prevent over-reliance on specific neurons, forcing the network to learn more robust and general feature representations, enhancing the generalization ability of the network, and avoiding overfitting. MHSA can enable the model to understand and represent the input data more carefully and accurately, and improve the overall feature extraction ability of the network. FFN performs deep non-linear transformation on the extracted features, which helps to mine deeper feature information.

[0081] It is worth mentioning that at the input end of MHSA, a unique method is used to process the complex spectrum, which is different from the traditional speech enhancement method of extracting phase information and amplitude information from the complex spectrum. In order to avoid the memory consumption caused by the extraction of phase information and amplitude information and the problem of difficult alignment between the two, the present invention adopts the following method:

[0082] First, the input noisy speech, that is, the noisy sample, is normalized and subjected to short-time Fourier transform STFT to obtain the complex spectrum . If it is the first iteration, also needs to be initialized to make = Integrate the output of the iteration with the complex spectrum of the noisy samples in accordance with a preset dimension to obtain a mixed complex spectrum , where represents the number of time domain frames, represents the Fourier length, represents the batch size. To match the input of the MHSA network, the mixed complex spectrum is reshaped into through the method of reshaping with a multi-dimensional tensor in the preset dimension. The dimension change of the multi-dimensional tensor is only to adapt to the network structure of the Transformer. Reshaping the multi-dimensional tensor according to a certain dimension is a relatively common processing method. Finally, it is restored to after passing through the Transformer module, that is the input to the Transformer module is output as .

[0083] The U-Net module consists of a convolutional autoencoder with skip connections. It is very effective in generation and segmentation tasks. Its structure is similar to a U shape and is divided into a downsampling module and an upsampling module. The downsampling module is used to extract features, reduce the resolution, and increase the number of feature channels. The upsampling module restores the resolution of the extracted features and precisely reconstructs them by combining them with the features from the corresponding encoder layers.

[0084] As Figures 3 to 6 shown, the network depth of the U-Net module designed in the present invention is 7. The U-Net module includes seven downsampling modules and seven upsampling modules.

[0085] Each downsampling module includes two branches. The first branch of the downsampling module includes two residual blocks and a downsampling block. The second branch of the downsampling module includes a downsampling operation module. The two residual blocks in the downsampling module are respectively defined as the first residual block and the second residual block.

[0086] Each upsampling module includes two branches. The first branch of the upsampling module includes three residual blocks and an upsampling block. The second branch of the upsampling module includes an upsampling operation module. The three residual blocks in the upsampling module are respectively defined as the third residual block, the fourth residual block, and the fifth residual block.

[0087] The processing flow of the signal in the U-Net module is as follows:

[0088] Take the output of the Transformer module as the input of the U-Net module and input it into the U-Net module First, the number of channels is increased to 128 through a convolutional network, obtaining .

[0089] Take as the input of the first branch and the second branch of the first downsampling module. In the first branch of the downsampling module, a set of eigenvalue is output after passing through the first residual block, the second residual block, and a downsampling block in sequence. In the second branch of the downsampling module, a set of eigenvalue is output after performing a downsampling operation. The outputs of the first branch and the second branch of the downsampling module are linearly fused to obtain the output of the first downsampling module, and the output of the current downsampling module is used as the input of the next downsampling module, and all the downsampling modules in the U-Net module are processed in sequence.

[0090] The output of the first downsampling module passes through a residual block and then outputs a set of eigenvalue, which is used as the input of the first branch and the second branch of the first upsampling module; the eigenvalue input into the first branch of the upsampling module is feature-stitched with the output of the downsampling block in the downsampling module at the corresponding level through a skip connection, and then input into the third residual block; the output of the third residual block is feature-stitched with the output of the second residual block in the downsampling module at the corresponding level through a skip connection, and then input into the fourth residual block; the output of the fourth residual block is feature-stitched with the output of the first residual block in the downsampling module at the corresponding level through a skip connection, and then input into the upsampling block. The output of the upsampling block is used as the output of the first branch of the upsampling module; the eigenvalue input into the second branch of the upsampling module is upsampled, and it is used as the output of the second branch of the upsampling module; the outputs of the first branch and the second branch of the upsampling module are linearly fused to obtain the output of the current upsampling module, and it is used as the input of the next upsampling module. All the upsampling modules in the U-Net module are processed in sequence, and the output of the last upsampling module is used as the output of the prediction network model of .

[0091] The difference in the number of residual blocks in the U-Net module is due to the fact that during the downsampling process, there are three features at each resolution level. These three features are obtained through the downsampling operation of the previous-level downsampling module and two identical convolutional strides, namely the three sets of eigenvalue output by the first residual block, the second residual block, and the downsampling block, covering adjacent feature information, and there is a strong correlation between them. Therefore, in the upsampling module, these three features need to be collected and incorporated into feature recovery. Therefore, the upsampling module needs to accumulate residuals from the downsampling module at the same resolution, which is more complex than the downsampling module. In addition, except for the different upsampling operation and downsampling operation in the upsampling block and the downsampling block, the other structures are the same.

[0092] Step 4. Considering the complex spectrum, phase spectrum, amplitude spectrum in the time domain, frequency domain, and time-frequency domain, as well as the time-domain signal and the measurement parameters of the time-domain signal, the prediction network model is trained by introducing a loss function. is trained.

[0093] The present invention introduces 5 loss functions, covering the complex spectrum, phase spectrum, amplitude spectrum in the time domain, frequency domain, and time-frequency domain, as well as the time-domain signal and the measurement parameters of the time-domain signal. For the complex spectrum loss, the present invention adopts the L2 loss :

[0094] (13)

[0095] where is the output of the prediction network model i.e., the prediction sample, and x represents the complex spectrum of the clean speech, i.e., the clean sample. This loss function is to find the difference between two complex spectra. represents the expected value. The loss function, generally, sums the differences between two vectors first and then takes the average, and this process is generally denoted by E.

[0096] Since the prediction network model only processes the complex spectrum and does not separately process the phase spectrum and amplitude spectrum, in order to balance the integrity of the feature information, the present invention introduces the phase spectrum loss function and the amplitude spectrum loss function , and the amplitude spectrum loss function is:

[0097] (14)

[0098] where and are respectively and 's amplitude information, represents the expected value.

[0099] Due to the periodicity of the trigonometric function, the phase is usually in the interval or . When calculating the signal phase information, it may exceed this limit, resulting in phase wrapping. The present invention uses the phase unwrapping function to first control the phase information within the interval, and then calculates the phase loss. The expression of

[0100] (15)

[0101] When calculating the phase loss, the present invention also introduces three kinds of phase losses of angles, namely the instantaneous phase loss 、instantaneous angular frequency loss , group delay loss . Since the instantaneous phase reflects the actual phase of the signal at each instant, the angular frequency determines the basic rate at which the signal phase changes with time, and the group delay reflects the time delay difference of different frequency components during transmission, only by comprehensively considering these three factors can the phase change of the signal be described comprehensively and accurately.

[0102] (16)

[0103] (17)

[0104] (18)

[0105] Among them, and are respectively and phase information of and respectively represent the differential operators along the time axis and the frequency axis, , , represent the expected value, and the total phase loss is the sum of the three components , , :

[0106] (19)

[0107] The ultimate goal of the prediction network model is to minimize the difference between the predicted samples and the clean samples, including the time domain range. It is necessary to first convert the complex spectrum into a time domain signal through the inverse short-time Fourier transform ISTFT. Therefore, the time domain loss function :

[0108] (20)

[0109] Among them, and are respectively and time domain signals of represents the expected value.

[0110] The metric loss is based on the objective speech quality assessment PESQ score, and this method has been proven to be very effective in the field of speech enhancement. The present invention uses the existing library functions:

[0111] (21)

[0112] Among them, A loss function representing the measurement parameters of a time-domain signal represents the speech quality determination index PESQ.

[0113] The overall loss function L is the sum of five loss functions, as follows:

[0114] (22)

[0115] Among them, 、 、 、 represent weight coefficients. In this embodiment = 0.1, = 0.0001, = 0.01, = 0.01. The speech signal features can be represented as time-frequency domain features and time-domain features. The loss functions of these two parts account for the largest proportion. However, since the input and output of the model in the method of the present invention are both based on the complex spectrum in the time-frequency domain, in this embodiment, the complex spectrum loss is set as the benchmark, and the weight of the complex spectrum loss function is 1. The time-domain loss function belongs to an auxiliary constraint, and its weight is less than 1, and it only needs to differ by one order of magnitude from the weight of the complex spectrum loss function. The phase loss and amplitude loss of the speech signal are two quantities obtained simultaneously based on the complex spectrum, and their weights are the same. Compared with the global features such as time-frequency domain features and time-domain features, they belong to detailed features. Therefore, the weight ratios of the phase spectrum loss function and the amplitude spectrum loss function also need to be less than the global features, and their weights and only need to differ by one order of magnitude from the weight of the time-domain loss function . As for the measurement loss, a single measurement loss only plays a guiding role and cannot occupy a large weight. A large weight will only guide the model to train for improving the index, rather than for the perfect approximation of the global features. Therefore, the weight of the loss function of the measurement parameters of the time-domain signal has the lowest proportion.

[0116] Step 5. Use the trained prediction network model to output a prediction sample for the input noisy speech, and obtain the enhanced speech according to the representation form of the solution of the stochastic differential equation.

[0117] In this embodiment, step 5 is specifically as follows:

[0118] Step 5.1. Perform normalization processing and STFT on the input noisy speech to obtain the complex spectrum Y of the noisy speech.

[0119] Step 5.2. Initialize to , initialize the time to t = T, and obtain the initial , , , ; Set the number of steps n and the step size N in the interval [0, T] t , and satisfy nN t = T.

[0120] In this embodiment, T is taken as 1, the number of steps is 50, and the step size is 0.02. Then the time is initialized to t = 1, and the number of steps 50 and the step size 0.02 are set in the interval [0, 1] to obtain the set of t [1, 0.98, …, 0.02, 0].

[0121] Step 5.3. Update the value of t to t = t - N t , and obtain the current and .

[0122] Input the parameters , Y, t into the trained prediction network model , and the prediction network model outputs the prediction sample .

[0123] According to the representation form of the solution of the differential equation shown in formula (12), obtain the solution of the differential equation in the iterative process , where is the complex spectrum Y of the noisy speech.

[0124] Step 5.4. Determine whether the current time t is 0; if so, use the at the current time as the final state of the reverse diffusion process, and go to Step 5.5; otherwise, set the at the current time as the initial state of the next iteration, i.e., , set the and at the current time to the and of the next iteration, i.e., , , and go to Step 5.3.

[0125] Step 5.5. Perform inverse STFT and denormalization processing on the of the final state to obtain the enhanced speech.

[0126] The purpose of the model is to transform the noisy samples into clean samples, so the noisy samples are known. In this embodiment, the reverse diffusion process is to gradually restore the noisy samples to clean samples by updating them 50 times, with the weights of the predicted samples, noisy samples, and the previous round of samples updated each time. During the iteration process, the parameters and are constantly changing. At the end of this round, is assigned to , and is assigned to for the convenience of the next round. This is the update of and . and remain fixed. The diffusion process is the process of iteratively updating . During the iteration process, is the of the previous round. After the iteration is completed, the finally obtained is the final state of the reverse diffusion process. Equation (12) is the solution of the reverse diffusion process. In Equation (12), is the final state of the forward diffusion process.

[0127] The voice enhancement algorithm of the present invention is based on the principle of the optimal path of the Schrödinger bridge, that is, to find the optimal path between the initial data distribution and the target data distribution, minimize the Kullback-Leibler divergence between the two distributions, construct the optimal transport path from the initial probability distribution to the target probability distribution, and convert the diffusion process into the solution process of a stochastic differential equation based on this principle, aiming to approximate the noisy samples to clean samples through the probability distribution. Different from the diffusion model that first learns the forward diffusion process from noisy samples to the Gaussian distribution, the voice enhancement algorithm proposed by the present invention directly learns the non-linear diffusion process from noisy samples to clean samples, thereby retaining more structural information of the initial samples.

[0128] The voice enhancement method of the present invention uses the Transformer module to capture the unique features of the time series signal. By introducing the Transformer network to preprocess the complex spectrum, it can directly use the complex spectrum as the input of the diffusion model, eliminating the cumbersome work of extracting the phase spectrum and amplitude spectrum from the complex spectrum and the memory overhead caused by the inverse transformation, and also avoiding the alignment problem between the phase spectrum and the amplitude spectrum. Then, the U-Net module is used to fuse multi-scale information. By introducing a 7-layer U-Net network and forming a convolutional network with the Transformer network, it is used to extract the predicted data. The present invention also takes into account the contributions of phase and amplitude to voice enhancement, adds phase loss and amplitude loss to the loss function, sets the ideal weights, and combines 5 loss functions covering the time domain, frequency domain, and time-frequency domain to gradually reduce the difference between the predicted samples and the clean samples.

[0129] The output speech obtained by the speech enhancement method of the present invention achieved evaluation results of 3.79, 4.72, 3.73, 4.36, and 0.95 in five objective evaluation indicators: PESQ speech quality perception evaluation, CSIG speech distortion, CBAK background noise interference, COVL overall speech quality, and STOI short-time objective intelligibility, respectively. Compared with the speech enhancement models in recent years, the best performance was achieved.

[0130] Figure 7 The entire training process in the embodiment of the present invention is shown through the loss curve of the training set. Figure 8 The entire training process in the embodiment of the present invention is shown through the loss curve of the validation set. Figure 7 and Figure 8 It is shown that during the entire training process, the loss function of the training set and the loss function of the validation set converge quickly to stability, which on the one hand reflects that the training of the network in the present invention is effective, and on the other hand reflects that the training of the network in the present invention is convergent. Figure 9 and Figure 10 The effectiveness of the training process is verified by testing the PESQ and STOI indicators during the training process. Figure 9 and Figure 10 It shows that the speech quality indicators PESQ and STOI continue to increase throughout the training process. Figure 11 The mainstream speech enhancement models in recent years are listed, where the horizontal axis is the year and the vertical axis is the PESQ score. Through comparison, it can be seen that the average PESQ score of the speech enhancement model in each year is increasing year by year, indicating the development trend of speech enhancement. The PESQ score of the enhanced speech output by the speech enhancement method of the present invention is the highest at present, which reflects the advanced nature of the speech enhancement method of the present invention.

[0131] Example 2

[0132] This embodiment 2 describes a diffusion model speech enhancement system based on Schrödinger bridge, which is based on the same inventive concept as the diffusion model speech enhancement method based on Schrödinger bridge in embodiment 1.

[0133] Specifically, the diffusion model speech enhancement system based on Schrödinger bridge includes the following modules:

[0134] The conversion module is used to convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation.

[0135] The stochastic differential equation solving module is used to determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, use the variance explosion to set the diffusion function and drift function in the stochastic differential equation, and then obtain the representation of the solution of the stochastic differential equation.

[0136] A model establishment module constructs a prediction network model through a Transformer module and a U-Net module. The prediction network model is used to output a prediction sample, that is, a process quantity involved in the representation of the solution of the stochastic differential equation related to the diffusion process.

[0137] A model training module is used to introduce loss functions of complex spectra, amplitude spectra, phase spectra, time-domain signals, and measurement parameters of time-domain signals to train the prediction network model.

[0138] And a voice enhancement module is used to utilize the trained prediction network model to output a prediction sample for the input noisy voice, and obtain enhanced voice according to the representation of the solution of the stochastic differential equation.

[0139] It should be noted that in the voice enhancement system based on the Schrödinger bridge diffusion model, the implementation processes of the functions and roles of each functional module are specifically described in the corresponding steps of the method in Embodiment 1, and will not be elaborated here.

[0140] Embodiment 3

[0141] This Embodiment 3 describes a computer device, which includes a memory and one or more processors.

[0142] An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the steps of the voice enhancement method based on the Schrödinger bridge diffusion model in the above-mentioned Embodiment 1.

[0143] In this embodiment, the computer device is any device or apparatus with data processing capabilities, which will not be elaborated here.

[0144] Embodiment 4

[0145] This Embodiment 4 describes a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the voice enhancement method based on the Schrödinger bridge diffusion model.

[0146] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device.

[0147] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to listing the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any person skilled in the art under the teaching of this specification fall within the substantial scope of this specification and should be protected by the present invention.

Claims

1. A Schrodinger bridge-based diffusion model speech enhancement method, characterized in that: The steps include: Step 1. Convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation; Step 2. Determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, use variance explosion to set the diffusion function and drift function in the stochastic differential equation, and then obtain the representation of the solution of the stochastic differential equation; Step 3. Build a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation; Step 4. Introduce the complex spectrum, amplitude spectrum, phase spectrum, time domain signal and the loss function of the measurement parameters of the time domain signal to train the prediction network model; Step 5. The trained prediction network model is used to output prediction samples for the input noisy speech, and the enhanced speech is obtained according to the representation of the solution of the stochastic differential equation; In step 3, a preprocessing operation is performed before inputting the Transformer module, and the noisy speech, i.e., the noisy sample, is normalized and short-time Fourier transformed (STFT) to obtain a complex spectrum. In the first iteration, x t Initialized to y, where x t represents the state at time t, B represents the batch size, T0 represents the number of time domain frames, and F represents the Fourier length; Will Complex spectrum of noisy samples Integrate to get the mixed complex spectrum The mixed complex spectrum x is reorganized by reorganizing the multidimensional tensor in the preset dimension. in Reshape The Transformer module includes a normalization layer, a multi-head self-attention mechanism MHSA, a fully connected network FFN, and a regularized Dropout layer. After entering the Transformer module, the output is In step 3, the U-Net module includes seven down-sampling modules and seven up-sampling modules; Each downsampling module includes two branches, the first branch of the downsampling module includes two residual blocks and a downsampling block, and the second branch of the downsampling module includes a downsampling operation module; Each upsampling module includes two branches, the first branch of the upsampling module includes three residual blocks and an upsampling block, and the second branch of the upsampling module includes an upsampling operation module; The signal processing flow in the U-Net module is as follows: The output x of the Transformer module out As the input of the U-Net module, input x in the U-Net module out First, the number of channels is increased to 128 through the convolutional network, and we get The output obtained after X is processed by seven downsampling modules and seven upsampling modules in the U-Net module is used as the prediction network model F θ The output x θ .

2. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 1, characterized in that: The step 1 is specifically as follows: For the speech enhancement process, the stochastic differential equation modeling the forward diffusion process is: dx t =f(x t )dt+g(t)dw; Among them, x t represents the state at time t, f(x t ) is the drift function, g(t) is the diffusion function, and w is the standard Wiener process; The stochastic differential equation modeling the back diffusion process is: in, is the inverse Wiener process, ▽ represents the gradient operator, ▽logp t represents the probability density function p t The logarithmic gradient of .

3. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 2, characterized in that: The step 2 is specifically as follows: The Schrödinger bridge is defined as path p and reference path p ref The Kullback-Leibler divergence between KL The minimization problem is: Wherein, T represents the time symbol; D KL The solution to the minimization problem is given by a pair of symmetric forward and backward stochastic differential equations: dx t =[f(x t )+g 2 (t)▽logψ t ]dt+g(t)dw; Among them, ▽logψ t and is the nonlinear drift term, ψ t and denote the forward and reverse drift functions, respectively; Get the representation x of the solution to the stochastic differential equation t for: Among them, α s , σ s , represents the initial parameters, α t represents the drift accumulation in [0, t], α T represents the drift accumulation in [0,T]; σ t represents the diffusion accumulation in [0, t], σ T represents the diffusion accumulation in [0,T], x s represents the initial state of each iteration, x T represents the state at time t = T, x θ Process quantities involving diffusion processes in the representation of the process state, i.e., the solution of the stochastic differential equation; Using the variance explosion solution, set Where c and k are hyperparameters, c and k control the diffusion amplitude and rate respectively, f(x t )=0, so α t =α T =1, 4. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 3, characterized in that: The step 4 is specifically as follows: For the complex spectrum loss, L2 loss L is used com : Among them, x θ is the prediction network model F θ The output of is the predicted sample, x represents the complex spectrum of clean speech, i.e., the clean sample. Indicates expected value; Introducing the amplitude spectrum loss function L mag : in, and x M x θ and the magnitude information of x, Indicates expected value; Using the phase unwrapping function f AW (t) Control the phase information in the interval [0, π], f AW The expression of (t) is: Calculate the instantaneous phase loss L IP , instantaneous angular frequency loss L IAF , group delay loss L GD : in, and x P They are x θ and x, ΔDT and ΔDF represent the differential operators along the time axis and frequency axis respectively. Indicates expected value; The total phase spectrum loss function L pha There are three components L IP , L IAF , L GD sum: L pha =L IP +L IAF +L GD ; Time domain loss function L dom for: in, and They are x θ and the time domain signal of x, Indicates expected value; The loss metric is based on the objective speech quality assessment PESQ score: L met =L torch_pesq.PesqLoss ; Among them, L met The loss function representing the metric parameter of the time domain signal, L torch_pesq.PesqLoss It represents the speech quality judgment index PESQ; The overall loss function L is obtained as: L=L com +λ1L dom +λ2L met +λ3L mag +λ4L pha ; Among them, λ1, λ2, λ3, and λ4 represent weight coefficients.

5. The Schrodinger bridge-based diffusion model speech enhancement method according to claim 4, characterized in that: The step 5 is specifically as follows: Step 5.

1. Perform normalization and STFT on the input noisy speech to obtain the complex spectrum Y of the noisy speech; Step 5.

2. Set x s Initialize to x s =Y, x t Initialize to x t =x s , initialize the time to t = T, and obtain the initial σ s , σ T , α s , α T ; Set the number of steps n and the step length N in the interval [0,T] t , and satisfy nN t =T; Step 5.

3. Update the value of t to t = tN t , and find the current σ t and α t ; The parameter x t , Y, t are input into the trained prediction network model F θ In the prediction network model, the prediction sample x is output θ ; According to the representation of the solution of the stochastic differential equation, where x T is the complex spectrum Y of the noisy speech, and the solution of the differential equation in the iterative process is obtained and used to update the current time x t ; Step 5.

4. Determine whether the current time t is 0; if so, set the current time x t As the final state of the back diffusion process, go to step 5.5; otherwise, set the current x t Set as the initial state of the next iteration, that is, x s =x t , and the current σ t and α t Assign the value to the next iteration σ s and α s , that is, s =σ t , α s =α t , then go to step 5.3; Step 5.

5. The final state x of the back diffusion process t Perform inverse STFT and denormalization processing to obtain enhanced speech.

6. A Schrodinger bridge-based diffusion model speech enhancement system for implementing the Schrodinger bridge-based diffusion model speech enhancement method according to claim 1, characterized in that: The Schrodinger bridge-based diffusion model speech enhancement system includes: A conversion module, used to convert the diffusion process of the diffusion model into the solution process of the stochastic differential equation; The stochastic differential equation solving module is used to determine the stochastic differential equation based on the theoretical principle of Schrödinger bridge, set the diffusion function and drift function in the stochastic differential equation by using variance explosion, and then obtain the representation of the solution of the stochastic differential equation; The model building module builds a prediction network model through the Transformer module and the U-Net module. The prediction network model is used to output the prediction sample, that is, the process quantity involved in the diffusion process in the representation of the solution of the stochastic differential equation; A model training module is used to introduce complex spectrum, amplitude spectrum, phase spectrum, time domain signal and loss function of measurement parameters of time domain signal to train the prediction network model; And a speech enhancement module is used to use the trained prediction network model to output prediction samples for the input noisy speech, and obtain enhanced speech according to the representation of the solution of the stochastic differential equation.

7. A computer device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the steps of the Schrödinger bridge-based diffusion model speech enhancement method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the Schrödinger bridge-based diffusion model speech enhancement method as described in any one of claims 1 to 5 are implemented.

Images