Universal speech enhancement system and method based on decoupling diffusion model

Abstract

The invention discloses a general speech enhancement system and method based on a decoupling diffusion model, and the method comprises the following steps: carrying out the feature extraction of speech signals through short-time Fourier transform, the speech signals comprising clean speech signals and mixed speech signals, and defining and obtaining distorted speech residual signals; noise is added to the extracted clean voice, and a distorted voice residual signal and Gaussian noise are added to the clean voice through a Markov chain; setting a pre-training target function and pre-training the model, and pre-training the model through a consistency residual error estimation target loss function; setting a fine tuning target function with a minimum sampling target to perform fine tuning on the model; carrying out iterative denoising on the mixed voice, and carrying out iterative denoising on the mixed voice through an inverse transition probability to obtain a denoised clean voice; and reconstructing a time domain signal through inverse short-time Fourier transform.

Description

Technical Field

[0001] This invention belongs to the field of speech processing technology, and in particular relates to a general speech enhancement system and method based on a decoupled diffusion model. Background Technology

[0002] With the deep integration of artificial intelligence and the Internet of Things (IoT) technologies, intelligent voice technology has been widely applied in human-computer interaction, smart homes, and remote collaboration, profoundly reshaping the production and lifestyle of modern society. However, in practical applications, voice signals are inevitably subject to various interferences during acquisition, transmission, and reconstruction, such as environmental noise, channel distortion, and equipment nonlinear distortion, leading to a significant decline in voice quality. Especially in complex acoustic environments, multiple voice distortion phenomena often occur simultaneously, posing a severe challenge to voice enhancement technology.

[0003] Traditional multi-task speech enhancement methods typically employ a divide-and-conquer strategy, designing separate processing modules for different distortion types. This approach is ineffective when dealing with diverse distortion problems in complex scenarios, and often requires algorithm redesign when task requirements change, increasing development costs and limiting system scalability. In the last two years, the concept of general speech enhancement has emerged, marking a shift in the research paradigm towards unified modeling. General speech enhancement technology aims to build a universal enhancement model, achieving integrated processing of various speech enhancement tasks and improving the system's versatility and robustness. Compared to traditional speech enhancement techniques, general speech enhancement faces unprecedented challenges. The main challenges are reflected in the following three aspects: 1) Diversity of distortion types: General-purpose speech enhancement needs to address multiple distortion types simultaneously, such as additive noise, reverberation, and nonlinear distortion, which places higher technical demands on the construction of a unified enhancement model; 2) Complexity of multi-task processing: The design of a general-purpose speech enhancement system needs to consider multiple sub-tasks, such as speech denoising, dereverberation, and echo cancellation, significantly increasing the complexity of system design and optimization; 3) Limitations in constructing generalized models: Due to the difficulty in obtaining sufficient and representative diverse speech distortion signals in real-world application scenarios, constructing a robust general-purpose speech enhancement model faces significant challenges. These factors combined make the development of general-purpose speech enhancement technology far more difficult than traditional speech enhancement methods. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a general speech enhancement system and method based on a decoupled diffusion model, which decouples the traditional single denoising diffusion process into distorted speech residual diffusion and noise diffusion, thereby achieving integrated processing of multiple types of speech distortion by a single model.

[0005] The technical solution of this invention is implemented as follows:

[0006] A general speech enhancement method based on a decoupled diffusion model includes the following steps:

[0007] S1. Given an initial mixed speech signal and a clean speech signal, the distorted speech residual signal is defined as the difference between the mixed speech signal and the clean speech signal; the signal is then subjected to feature extraction using a short-time Fourier transform to obtain its frequency domain representation.

[0008] S2. In the frequency domain, the frequency domain distorted speech residual signal and Gaussian noise are continuously added to the frequency domain clean speech signal through a Markov chain to improve the degree of mixed speech distortion.

[0009] S3. Pre-train the model by estimating the target loss function using consistency residuals;

[0010] S4. Fine-tune the model by setting a fine-tuning objective function that minimizes the sampling target;

[0011] S5. The mixed speech is denoised iteratively using inverse transition probability to obtain the denoised clean speech.

[0012] S6. Reconstruct the time-domain signal using inverse short-time Fourier transform.

[0013] Furthermore, in step S1, specifically, x0(t) is defined as representing a clean speech signal, x res (t) represents the distorted speech residual signal; the mixed speech signal y(t) is represented as:

[0014] y(t)=x0(t)+x res (t);

[0015] Feature extraction was performed on the above signals using short-time Fourier transform to obtain the clean speech signal X0 in the frequency domain, the mixed speech signal Y in the frequency domain, and the distorted speech residual signal X in the frequency domain. res Representation format:

[0016] X res =Y-X0,

[0017] Furthermore, in S2, the decoupling diffusion process is defined to add noise to the clean speech signal in the frequency domain, specifically including: constructing a T-step Markov chain, where T is the total number of steps in the diffusion process; and respectively adding noise to the frequency domain distorted speech residual signal X. res Gaussian noise ∈ is added to the clean speech signal X0 in the frequency domain, resulting in the t-1 step frequency domain distorted speech residual signal X. t-1 Single-step noise addition to t-step frequency domain distortion speech residual signal X t The transition probability is defined as:

[0018]

[0019] Where q(·) represents the transition probability, and t represents the current step number in the diffusion process; α t Represents the residual coefficient of distorted speech; β t represents the Gaussian noise coefficient; I is the identity matrix.

[0020] Furthermore, based on the defined single-step noise addition process and reparameter re-parameter technique, X t It can be obtained directly from X0:

[0021]

[0022] The transition probability can be expressed as:

[0023]

[0024] Among them, hyperparameters and Each satisfies

[0025] When t = T, the coefficient At this point, the distribution is only related to the frequency domain mixed speech signal Y and the added Gaussian noise ∈. Control the Gaussian noise intensity during step T.

[0026] Furthermore, in S3, based on the fundamental principles of DDPM, the pre-training objective function is derived through variational inference:

[0027]

[0028] Among them, D KL [*||*] denotes the Kullback-Leibler (KL) divergence, q(X t-1 |X t ,X0,X res Let p be the target distribution. θ (X t-1 |X t ) indicates from X t To X t-1 The transition probability has a learnable parameter θ;

[0029] Define p θ (X t-1 |X t )for:

[0030] p θ (X t-1 |X t )=N(X t-1 μ θ (X t ,t),Σ θ (Xt ,t));

[0031] Where, μ θ (X t ,t) is the mean predicted by the model parameters θ, Σ θ (X t ,t) is the variance predicted by the model parameters θ;

[0032] According to Bayes' theorem, the target distribution q(X) t-1 |X t ,X0,X res ) is represented as:

[0033]

[0034] Through the target distribution q(X) t-1 |X t ,X0,X res To obtain X t-1 The distribution of X is simplified and rearranged into a distribution with respect to X. t-1 The form;

[0035] Wherein, C(X) t ,X0,X res ) is with X t-1 Irrelevant constants;

[0036] Further, the mean μ(X) of the target distribution is obtained. t ,t,X res ) and variance Σ(X t ,t):

[0037]

[0038] Then p θ (X t-1 |X t The mean μ θ (X t (t,Y) and variance Σ θ (X t ,t) is represented as:

[0039]

[0040] in, Predicted by a model with parameter θ, ∈ θ It is by We obtain: The KL divergence of two Gaussian distributions is simplified to the difference between their means, further simplifying the pre-training objective function to:

[0041]

[0042] According to Xt The formula obtained from sampling X0, ∈ θ Depend on Export, as ∈ θ Substituting the simplified pre-training objective function, we obtain the simplified original objective loss function:

[0043]

[0044] in, This represents the original objective loss function.

[0045] Furthermore, in step S3, the objective function for consistency residual estimation is set, specifically including randomly selecting two different time steps t and t′, and adding the residuals of different time steps to the clean frequency domain speech signal X0 to obtain the perturbed frequency domain distorted speech residual signal X. t and X t′ Then, the perturbed frequency-domain distorted speech residual signal is input into the residual network. Make a prediction and obtain the predicted residual. and Finally, the model is pre-trained using the consistency residual estimation objective loss function, which is defined as:

[0046]

[0047] in, This represents the objective loss function for consistency residual estimation.

[0048] Furthermore, in S4, the fine-tuning objective function for minimizing the sampling target is set. Specifically, for each training pair (X0, Y) of clean speech signal and mixed speech signal in the frequency domain, the pre-trained model is used to iteratively sample the mixed speech signal Y in the frequency domain to generate the predicted clean speech. The objective function for minimizing the sampling error in the fine-tuning phase is defined as minimizing the predicted clean speech. Distance from the true frequency domain clean speech signal X0:

[0049]

[0050] in, This represents the objective loss function that minimizes the sampling error.

[0051] Furthermore, in step S5, iterative denoising of the mixed speech specifically includes: iteratively denoising the frequency domain mixed speech signal Y using inverse transition probabilities to obtain a clean frequency domain speech signal X0, and representing it as p using a posterior distribution with parameter θ. θ (X0|Y):

[0052]

[0053] The initial distribution approximately follows a mean of X. T A Gaussian distribution with variance I is represented as p(X). T |Y)≈N(X T |Y,I);X T Let I be a clean speech signal in the frequency domain with a diffusion step number of T, and I be the identity matrix.

[0054] Furthermore, based on the reparameterization technique, when DDPM is used as the sampling strategy, X t-1 It can be determined by the transition probability p θ (X t-1 |X t The mean μ θ (X t (t,Y) and variance Σ θ (X t The data is obtained by sampling (t), and is specifically expressed as follows:

[0055]

[0056] Where, ∈ * This represents random Gaussian noise. To accelerate the sampling process, a deterministic sampling strategy from DDIM is adopted. This strategy simplifies the calculation process by eliminating random noise terms. Specifically, based on the reparameterization technique, the speech signal X at the current diffusion step is... t Obtain the speech signal X from the previous diffusion step. t-1 The calculation method is simplified as follows:

[0057]

[0058] The sampling process begins at t=T, employing the deterministic sampling strategy of DDIM, iterating backwards according to the diffusion steps, starting from X. T Restore to A clean speech signal was eventually recovered. Where k represents the number of steps skipped in the iteration.

[0059] The general speech enhancement system based on the decoupled diffusion model applies the general speech enhancement method based on the decoupled diffusion model described above.

[0060] Compared with the prior art, the present invention achieves the following beneficial effects:

[0061] This invention provides a general speech enhancement system and method based on a decoupled diffusion model. First, STFT feature extraction is performed on the speech signal. Second, a decoupled diffusion process is constructed to decouple various distortions in the speech signal into composite distortion residuals and Gaussian noise. A Markov chain is then used to superimpose the speech distortion residuals and Gaussian noise into clean speech, achieving unified modeling of multiple types of distortions such as additive noise, reverberation, and nonlinear distortion. Subsequently, a two-stage training strategy is employed: first, the model is pre-trained based on a designed Consistency Residual Estimation (CRE) objective function; then, fine-tuning is performed by minimizing the sampling error using the SEM objective function, effectively reducing sampling bias in real-world scenarios. After training, the model is used to perform denoising and reconstruction on mixed speech, finally reconstructing clean speech in the time domain using an inverse short-time Fourier transform. This invention achieves unified modeling of multiple speech distortion types through diffusion process decoupling and a two-stage optimization mechanism, significantly improving the model's robustness in complex real-world scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart of a general speech enhancement method based on a decoupled diffusion model provided in an embodiment of the present invention;

[0063] Figure 2 This describes the diffusion and sampling process of the general speech enhancement method based on a decoupled diffusion model provided in this embodiment of the invention.

[0064] Figure 3 This is the two-stage training framework for a general speech enhancement method based on a decoupled diffusion model provided in this embodiment of the invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0066] Example

[0067] like Figures 1 to 3 A general speech enhancement method based on a decoupled diffusion model.

[0068] The meanings of some English abbreviations in this embodiment are explained as follows:

[0069] STFT Short-Time Fourier Transform;

[0070] DDPM (Denoising Diffusion Probabilistic Models) is a generative model based on Markov chains that compresses a signal into Gaussian noise by progressively adding noise, and then gradually restores the original signal through a denoising process.

[0071] DDIM (Denoising Diffusion Implicit Models) is a generative method based on diffusion models that significantly improves generation efficiency by refining the sampling process while maintaining the same training objective as DDPM.

[0072] DDIM reduces the number of sampling steps by introducing a non-Markov diffusion process instead of the Markov chain in traditional DDPM. During training, DDIM and DDPM use the same objective, but during the generation process, DDIM accelerates generation by merging the sampling operations of intermediate steps, combining the Gaussian noise that originally required multiple steps into a single step of sampling.

[0073] CRE (Consistency Residual Estimation) is the core objective loss function designed for the decoupling diffusion process during the model pre-training phase of this invention. Its core idea is to simultaneously optimize the residual prediction accuracy at a single time point and the residual consistency across multiple time points.

[0074] SEM (Sampling Target Minimization) is the objective function used to optimize actual sampling performance during the model fine-tuning stage of this invention. Its core logic is to directly align the model's final output with real, clean speech.

[0075] MSE stands for Mean Squared Error.

[0076] Specifically, the following steps are included:

[0077] S1. Given an initial mixed speech signal and a clean speech signal, the distorted speech residual signal is defined as the difference between the mixed speech signal and the clean speech signal; the signal is then subjected to feature extraction using a short-time Fourier transform to obtain its frequency domain representation.

[0078] Specifically, S1 includes setting x0(t) to represent a clean speech signal, x res (t) represents the distorted speech residual signal, which includes additive noise, convolutional reverberation, and nonlinear distortion; the mixed speech signal y(t) is represented as their superposition:

[0079] y(t)=x0(t)+x res (t);

[0080] Feature extraction was performed on the above signals using short-time Fourier transform to obtain the clean speech signal X0 in the frequency domain, the mixed speech signal Y in the frequency domain, and the distorted speech residual signal X in the frequency domain. res Representation format:

[0081] X res =Y-X0,

[0082] S2. In the frequency domain, the frequency domain distorted speech residual signal and Gaussian noise are continuously added to the frequency domain clean speech signal through a Markov chain, making the degree of mixed speech distortion higher and higher.

[0083] In S2, the decoupling diffusion process is defined to add noise to the clean speech signal in the frequency domain. Specifically, it includes constructing a T-step Markov chain, where T is the total number of steps in the diffusion process; and converting the frequency domain distorted speech residual signal X into... res Gaussian noise ∈ is added to the clean speech signal X0 in the frequency domain, resulting in the t-1 step frequency domain distorted speech residual signal X. t-1 Single-step noise addition to t-step frequency domain distortion speech residual signal X t The transition probability is defined as:

[0084]

[0085] Where q(·) represents the transition probability, and t represents the current step number in the diffusion process; α t Represents the residual coefficient of distorted speech; β t represents the Gaussian noise coefficient; I is the identity matrix.

[0086] Based on the defined single-step noise addition process and reparameter re-parameter technique, X t It can be obtained directly from X0:

[0087]

[0088] The transition probability can be expressed as:

[0089]

[0090] Among them, hyperparameters and Each satisfies

[0091] Hyperparameters are parameters that are set manually before model training and are not learned through the training process.

[0092] When t = T, the coefficient At this point, the distribution is only related to the frequency domain mixed speech signal Y and the added Gaussian noise ∈. Control the Gaussian noise intensity during step T.

[0093] S3. Pre-train the model by estimating the target loss function using consistency residuals;

[0094] In S3, based on the fundamental principles of DDPM, the pre-training objective function is derived through variational inference:

[0095]

[0096] Among them, D KL [*||*] denotes the Kullback-Leibler (KL) divergence, q(X t-1 |X t ,X0,X res Let p be the target distribution. θ (X t-1 |X t ) indicates from X t To X t-1 The transition probability has a learnable parameter θ;

[0097] Define p θ (X t-1 |X t )for:

[0098] p θ (X t-1 |X t )=N(X t-1 μ θ (X t ,t),Σ θ (X t ,t));

[0099] Where, μ θ (X t ,t) is the mean predicted by the model parameters θ, Σ θ (X t ,t) is the variance predicted by the model parameters θ;

[0100] According to Bayes' theorem, the target distribution q(X) t-1 |X t ,X0,X res ) is represented as:

[0101]

[0102] Through the target distribution q(X) t-1 |X t ,X0,X res To obtain X t-1 The distribution of X is simplified and rearranged into a distribution with respect to X. t-1 The form;

[0103] Wherein, C(X)t ,X0,X res ) is with X t-1 Irrelevant constants;

[0104] Further, the mean μ(X) of the target distribution is obtained. t ,t,X res ) and variance Σ(X t ,t):

[0105]

[0106] Then p θ (X t-1 |X t The mean μ θ (X t (t,Y) and variance Σ θ (X t ,t) is represented as:

[0107]

[0108] in, Predicted by a model with parameter θ, ∈ θ It is by We obtain: The KL divergence of two Gaussian distributions is simplified to the difference between their means, further simplifying the pre-training objective function to:

[0109]

[0110] According to X t The formula obtained from sampling X0, ∈ θ Depend on Export, as ∈ θ Substituting the simplified pre-training objective function, we obtain the simplified original objective loss function:

[0111]

[0112] in, This represents the original objective loss function.

[0113] In step S3, the objective function for consistency residual estimation is set, specifically including randomly selecting two different time steps t and t′, and adding the residuals of different time steps to the clean frequency domain speech signal X0 to obtain the perturbed frequency domain distorted speech residual signal X. t and X t′ Then, the perturbed frequency-domain distorted speech residual signal is input into the residual network. Make a prediction and obtain the predicted residual. and Finally, the model is pre-trained using the consistency residual estimation objective loss function, which is defined as:

[0114]

[0115] in, This represents the objective loss function for consistency residual estimation.

[0116] A pre-training objective function is designed to pre-train the model. During pre-training, the original objective function only optimizes a single time point, leading to error accumulation during sampling. Therefore, a consistent residual estimation objective loss function is designed to simultaneously optimize the residual estimation at a single time point while maintaining consistency across estimated residuals at different time points. This approach improves the accuracy of residual estimation across different time steps during sampling, thereby reducing accumulated errors during the sampling process.

[0117] S4. Fine-tune the model by setting a fine-tuning objective function that minimizes the sampling target;

[0118] In step S4, a fine-tuning objective function for minimizing the sampling target is set. Specifically, for each training pair (X0, Y) of clean speech signal and mixed speech signal in the frequency domain, the pre-trained model is used to iteratively sample the mixed speech signal Y in the frequency domain to generate the predicted clean speech. The objective function for minimizing the sampling error in the fine-tuning phase is defined as minimizing the predicted clean speech. Distance from the true frequency domain clean speech signal X0:

[0119]

[0120] in, This represents the objective loss function that minimizes the sampling error.

[0121] A fine-tuning objective function was designed to fine-tune the model. During the fine-tuning phase, a sampling error minimization (SEM) objective loss function was designed to optimize the model's performance during real-world sampling. The core of this objective function is to minimize the distance between the predicted clean speech obtained through sampling and the real clean speech, thereby significantly reducing the errors generated during the sampling process.

[0122] S5. The mixed speech is denoised iteratively using inverse transition probability to obtain the denoised clean speech.

[0123] In step S5, iterative denoising of the mixed speech specifically includes: iteratively denoising the frequency domain mixed speech signal Y using inverse transition probabilities to obtain a clean frequency domain speech signal X0, and representing it as a posterior distribution with parameter θ as p. θ (X0|Y):

[0124]

[0125] The initial distribution approximately follows a mean of X. T A Gaussian distribution with variance I is represented as p(X). T |Y)≈N(X T |Y,I);X T Let I be a clean speech signal in the frequency domain with a diffusion step number of T, and I be the identity matrix.

[0126] Based on the reparameterization technique, when DDPM is used as the sampling strategy, X t-1 It can be determined by the transition probability p θ (X t-1 |X t The mean μ θ (X t (t,Y) and variance Σ θ (X t The data is obtained by sampling (t), and is specifically expressed as follows:

[0127]

[0128] Where, ∈ * This represents random Gaussian noise. To accelerate the sampling process, a deterministic sampling strategy from DDIM is adopted. This strategy simplifies the calculation process by eliminating random noise terms. Specifically, based on the reparameterization technique, the speech signal X at the current diffusion step is... t Obtain the speech signal X from the previous diffusion step. t-1 The calculation method is simplified as follows:

[0129]

[0130] The sampling process begins at t=T, employing the deterministic sampling strategy of DDIM, iterating backwards according to the diffusion steps, starting from X. T Restore to A clean speech signal was eventually recovered. Where k represents the number of steps skipped in the iteration.

[0131] S6. Reconstruct the time-domain signal using inverse short-time Fourier transform.

[0132] Finally, the inverse short-time Fourier transform (ISTFT) is used to... Reconstructing the time domain signal

[0133] The general speech enhancement system based on the decoupled diffusion model applies the general speech enhancement method based on the decoupled diffusion model as described above.

[0134] Inspired by the significant success of residual diffusion models in image restoration, this invention designs a general speech enhancement system based on a decoupled diffusion model. This method decouples the traditional single denoising diffusion process into residual diffusion of distorted speech and noise diffusion. Unlike traditional diffusion models that treat noise as a whole, the mechanism of decoupling residuals and noise can accurately and uniformly model various types of distortion in real-world scenarios, thereby achieving integrated processing of multiple types of speech distortion by a single model. Furthermore, the two-stage training strategy proposed in this invention effectively suppresses error accumulation during the real sampling process, further enhancing the model's robustness in real-world scenarios.

[0135] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A general speech enhancement method based on a decoupled diffusion model, characterized in that, Includes the following steps: S1. Given an initial mixed speech signal and a clean speech signal, the distorted speech residual signal is defined as the difference between the mixed speech signal and the clean speech signal; the signal is then subjected to feature extraction using a short-time Fourier transform to obtain its frequency domain representation. S2. In the frequency domain, the frequency domain distorted speech residual signal and Gaussian noise are continuously added to the frequency domain clean speech signal through a Markov chain to improve the degree of mixed speech distortion. S3. Pre-train the model by estimating the target loss function using consistency residuals; S4. Fine-tune the model by setting a fine-tuning objective function that minimizes the sampling target; S5. The mixed speech is denoised iteratively using inverse transition probability to obtain the denoised clean speech. S6. Reconstruct the time-domain signal using inverse short-time Fourier transform.

2. The general speech enhancement method based on the decoupled diffusion model according to claim 1, characterized in that, Specifically, S1 includes setting x0(t) to represent a clean speech signal, x res (t) represents the distorted speech residual signal; the mixed speech signal y(t) is represented as: y(t)=x0(t)+x res (t); Feature extraction was performed on the above signals using short-time Fourier transform to obtain the clean speech signal X0 in the frequency domain, the mixed speech signal Y in the frequency domain, and the distorted speech residual signal X in the frequency domain. res Representation format: X res =Y-X0。 3. The general speech enhancement method based on the decoupled diffusion model according to claim 1, characterized in that, In S2, the decoupling diffusion process is defined to add noise to the clean speech signal in the frequency domain. Specifically, it includes constructing a T-step Markov chain, where T is the total number of steps in the diffusion process; and converting the frequency domain distorted speech residual signal X into... res Gaussian noise ∈ is added to the clean speech signal X0 in the frequency domain, resulting in the t-1 step frequency domain distorted speech residual signal X. t-1 Single-step noise addition to t-step frequency domain distortion speech residual signal X t The transition probability is defined as: Where q(·) represents the transition probability, t represents the current step number of the diffusion process, and α t Represents the residual coefficient of distorted speech; β t represents the Gaussian noise coefficient; I is the identity matrix.

4. The general speech enhancement method based on the decoupled diffusion model according to claim 3, characterized in that, In S2, according to the defined single-step noise addition process and reparameter repetition technique, X t It can be obtained directly from X0: The transition probability can be expressed as: Among them, hyperparameters and Each satisfies When t = T, the coefficient At this point, the distribution is only related to the frequency domain mixed speech signal Y and the added Gaussian noise ∈. Control the Gaussian noise intensity during step T.

5. The general speech enhancement method based on the decoupled diffusion model according to claim 4, characterized in that, In S3, based on the fundamental principles of DDPM, the pre-training objective function is derived through variational inference: Among them, D KL [*||*] denotes the Kullback-Leibler (KL) divergence, q(X t-1 |X t ,X0,X res Let p be the target distribution. θ (X t-1 |X t ) indicates from X t To X t-1 The transition probability has a learnable parameter θ; Define p θ (X t-1 |X t )for: p θ (X t-1 |X t )=N(X t-1 ;μ θ (X t ,t),Σ θ (X t ,t)); Where, μ θ (X t ,t) is the mean predicted by the model parameters θ, Σ θ (X t ,t) is the variance predicted by the model parameters θ; According to Bayes' theorem, the target distribution q(X) t-1 |X t ,X0,X res ) is represented as: Through the target distribution q(X) t-1 |X t ,X0,X res To obtain X t-1 The distribution of X is simplified and rearranged into a distribution with respect to X. t-1 The form; Wherein, C(X) t ,X0,X res ) is with X t-1 Irrelevant constants; Further, the mean μ(X) of the target distribution is obtained. t ,t,X res ) and variance Σ(X t ,t): Then p θ (X t-1 |X t The mean μ θ (X t (t,Y) and variance Σ θ (X t ,t) is represented as: in, Predicted by a model with parameter θ, ∈ θ It is by We obtain: The KL divergence of two Gaussian distributions is simplified to the difference between their means, further simplifying the pre-training objective function to: According to X t The formula obtained from sampling X0, ∈ θ Depend on Export, as ∈ θ Substituting the simplified pre-training objective function, we obtain the simplified original objective loss function: in, This represents the original objective loss function.

6. The general speech enhancement method based on the decoupled diffusion model according to claim 5, characterized in that, In step S3, the objective function for consistency residual estimation is set, specifically including randomly selecting two different time steps t and t′, adding the residuals of different time steps to the clean frequency domain speech signal X0 respectively, and obtaining the perturbed frequency domain distorted speech residual signal X. t and X t′ Then, the perturbed frequency-domain distorted speech residual signals are input into the residual networks respectively. Make a prediction and obtain the predicted residual. and Finally, the model is pre-trained using the consistency residual estimation objective loss function, which is defined as: in, This represents the objective loss function for consistency residual estimation.

7. The general speech enhancement method based on the decoupled diffusion model according to claim 6, characterized in that, In step S4, a fine-tuning objective function for minimizing the sampling target is set. Specifically, for each training pair (X0, Y) of clean speech signal and mixed speech signal in the frequency domain, the pre-trained model is used to iteratively sample the mixed speech signal Y in the frequency domain to generate the predicted clean speech. The objective function for minimizing the sampling error in the fine-tuning phase is defined as minimizing the predicted clean speech. Distance from the true frequency domain clean speech signal X0: in, This represents the objective loss function that minimizes the sampling error.

8. The general speech enhancement method based on the decoupled diffusion model according to claim 3, characterized in that, In step S5, iterative denoising of the mixed speech specifically includes iteratively denoising the frequency domain mixed speech signal Y using inverse transition probabilities to obtain a clean frequency domain speech signal X0, which is represented by a posterior distribution with parameter θ as p. θ (X0|Y): The initial distribution approximately follows a mean of X. T A Gaussian distribution with variance I is represented as p(X). T |Y)≈N(X T |Y,I);X T Let I be a clean speech signal in the frequency domain with a diffusion step number of T, and I be the identity matrix.

9. The general speech enhancement method based on the decoupled diffusion model according to claim 8, characterized in that, In S5, based on the reparameterization technique, when DDPM is used as the sampling strategy, X t-1 It can be determined by the transition probability p θ (X t-1 |X t The mean μ θ (X t (t,Y) and variance Σ θ (X t The data is obtained by sampling (t), and is specifically expressed as follows: Where, ∈ * This represents random Gaussian noise. To accelerate the sampling speed, a deterministic sampling strategy from DDIM is adopted, which simplifies the calculation process by eliminating random noise terms. Specifically, based on reparameterization techniques, the speech signal X at the current diffusion step is... t Obtain the speech signal X from the previous diffusion step. t-1 The calculation method is simplified as follows: The sampling process begins at t=T, employing the deterministic sampling strategy of DDIM, iterating backwards according to the diffusion steps, starting from X. T Restore to A clean speech signal was eventually recovered. Where k represents the number of steps skipped in the iteration.

10. A general speech enhancement system based on a decoupled diffusion model, characterized in that, The general speech enhancement method based on the decoupled diffusion model as described in any one of claims 1 to 9 is applied.

Images