A speech enhancement method based on distribution enhanced diffusion model

Abstract

The application discloses a speech enhancement method based on a distribution enhancement diffusion model, relates to the technical field of single-channel speech enhancement, and constructs a unified interpolation framework for speech enhancement under a diffusion model. The core lies in designing a new type of generalized random differential equation disturbance kernel to unify existing interpolation strategies and reveal the essential role of the distribution enhancement mechanism. The method is based on mean value interpolation, proposes a general interpolation formula, covers various existing variants, theoretically determines the role of interpolation in data enhancement, and improves the effectiveness and stability of speech processing.

Description

Technical Field

[0001] This invention relates to the field of single-channel speech enhancement technology, and more specifically to a speech enhancement method based on a distributed enhancement diffusion model. Background Technology

[0002] Speech enhancement (SE) aims to recover clear speech from noisy audio and is widely used in scenarios such as voice communication, intelligent assistants, and speech recognition preprocessing, especially in complex or non-stationary noise environments. Traditional methods rely on signal processing techniques such as frequency domain filtering and spectral subtraction, but their effectiveness is limited when multiple types of noise are mixed. With the development of deep learning, modern speech enhancement methods have been continuously optimized in terms of speech quality, real-time performance, and adaptability, driving the rapid evolution of related technologies.

[0003] In recent years, diffusion models have been introduced into speech enhancement tasks due to their powerful generative capabilities. The basic principle is to diffuse clean speech into noisy data by progressively adding Gaussian noise, and then use a neural network to learn the anti-diffusion process to complete denoising. To improve training stability, mean interpolation strategies are widely used to construct intermediate states between clean and noisy speech. Early methods were mostly based on discrete-time diffusion models; subsequent research extended them to a continuous-time diffusion framework, combining drift term optimization of stochastic differential equations, exhibiting stronger adaptability and generalization effects under complex noise conditions. Although current methods have made some progress, the role of mean interpolation in diffusion models still lacks a unified understanding and theoretical support. Design strategies often rely on experience, making flexible adjustment difficult and limiting their application in different scenarios.

[0004] Therefore, it is necessary to propose a speech enhancement method based on a distribution-enhanced diffusion model to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to address the current lack of a unified understanding and theoretical support for the role of mean interpolation in diffusion models, the fact that design strategies rely heavily on experience and are difficult to adjust flexibly, which limits its application in different scenarios.

[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0007] A speech enhancement method based on a distributed enhancement diffusion model includes the following steps:

[0008] S1. Construct the relationship between the noisy speech signal c, the clean speech signal a, and the noise n:

[0009] c = a + n;

[0010] S2. Based on the diffusion model, a noisy speech signal a is generated by progressively adding Gaussian noise to the clean speech signal a through a forward process.t Its forward process can be represented by a stochastic differential equation:

[0011] da t = f(a, t)dt+g(t)dw;

[0012] Where f(a,t) is the drift coefficient, g(t) is the diffusion coefficient, and dw is the standard Wiener process;

[0013] S3. In the reverse process, the scoring function is learned through a neural network. To reconstruct a clean speech signal. Furthermore, the closed-form solution of the forward process at any time step t is:

[0014] a t =α t a0+σ t ε;

[0015] Among them, a t β is the scaling factor for the clean signal a0. t ε is the noise scaling function, where ε is Gaussian noise.

[0016] Furthermore, using a mean interpolation strategy, the noisy speech signal c and the clean speech signal a are weighted and averaged. The forward generalized stochastic differential equation for this is:

[0017]

[0018] Where, η t Time-dependent perturbation kernel coefficients This is a disturbance term.

[0019] Furthermore, the noisy sample representation of the mean interpolation strategy at any time step t is as follows:

[0020] a t =α t [η t a0+(1-η t )c]+β t ε.

[0021] Furthermore, the mean interpolation is extended from the perspective of distribution enhancement by incorporating both noise n and Gaussian noise ε into the transformation process, as expressed in the following expression:

[0022]

[0023] Among them, κ t This is a coefficient that varies with time.

[0024] Furthermore, it further includes distribution enhancement steps:

[0025] Decouple noise n from noisy speech signal to obtain pure noise signal;

[0026] Randomly shuffle the noise signal to generate perturbed noise. ;

[0027] The noise after mixing Clean speech a0 is recombined to generate new noisy speech samples.

[0028] Furthermore, the forward process of distribution enhancement is represented as:

[0029]

[0030] in, This is the noise signal after rinsing.

[0031] Furthermore, the generalized stochastic differential equation for the reverse process is expressed as:

[0032]

[0033] in, By neural network μ θ predict.

[0034] Furthermore, the method is applicable to single-channel speech enhancement tasks for recovering a clean speech signal a from a noisy speech input c.

[0035] Furthermore, the distribution enhancement method is executed during the training phase, and the speech enhancement results are generated using the reverse process of the standard diffusion model during the inference phase.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. This invention, by elevating mean interpolation to a diffusion kernel at the distribution enhancement level and combining it with a decoupled noise shuffling strategy, not only theoretically constructs a unified and comprehensive interpolation framework, but also significantly improves the model's speech recovery capability in complex noise environments in practice.

[0038] 2. The method of this invention also has significant advantages in practical application. Since noise shuffling only occurs during the training phase, and the inference phase still uses the standard inverse SDE generation process, no additional computational overhead or complex noise manipulation is required. This means that it can be seamlessly compatible with existing diffusion model accelerated sampling techniques during deployment, ensuring the feasibility of real-time or near-real-time speech enhancement.

[0039] 3. This invention constructs a generalized SDE perturbation kernel that covers various existing mean interpolation variants by introducing unified interpolation coefficients into two diffusion kernels: standard variance explosion and variance preservation. This innovation not only preserves the model's accurate simulation of the clean speech-noise mixture mechanism but also provides a flexible and controllable parameterization path for noise reconstruction. Experimental results show that the diffusion model using the method of this invention reaches or exceeds the current state-of-the-art levels in several mainstream speech quality and intelligibility metrics, including PESQ, STOI, CSIG, CBAK, and COVL. Attached Figure Description

[0040] Figure 1 The diagram below illustrates the principle of the diffusion model-based speech enhancement method based on distribution enhancement, as provided in this embodiment of the invention.

[0041] Figure 2 This is the speech predictor used in this embodiment. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] SDE stands for Stochastic Differential Equation, used to construct continuous-time diffusion generative models. Compared to discrete diffusion models, SDE offers a more flexible modeling approach. For related theory and implementation, please refer to the literature [Song Y et al., Score-based generative modeling through stochastic differential equations, ICLR 2021].

[0044] Please see Figure 1 This invention presents a speech enhancement method based on a distributed enhancement diffusion model. The aim is to construct a unified interpolation framework for speech enhancement under a diffusion model. The core of this method lies in designing a novel generalized stochastic differential equation (SDE) perturbation kernel to unify existing interpolation strategies and reveal its essential role as a distributed enhancement mechanism. Based on mean interpolation, this method proposes a universal interpolation formula that covers various existing variants, theoretically clarifying the role of interpolation in data enhancement and improving the effectiveness and stability of speech processing.

[0045] Building upon this foundation, this invention further introduces a distribution enhancement method with noisy shuffling. By recombining decoupled noise components to generate new clean-noisy signal pairs, the training distribution is effectively expanded, improving model performance. The proposed SDE perturbation kernel not only unifies and generalizes the interpolation strategy but also strengthens the model's adaptability to complex noise, driving speech enhancement technology towards greater robustness and generalization ability.

[0046] The goal of single-channel speech enhancement is to improve the quality of noisy speech signals. Assume the noisy signal *c* represents a mixture of clean speech *a* and noise *n*. This relationship can be expressed mathematically as follows:

[0047] c = a + n

[0048] Therefore, the goal of speech enhancement (SE) is to recover a clean speech signal a from a noisy speech input c. Discriminative methods directly learn the mapping from noisy speech input c to clean speech signal a, relying on large amounts of data and having limited generalization ability. To improve the generalization ability of the model, diffusion-based speech enhancement models introduce varying degrees of Gaussian noise into the forward process to degrade the clean speech, and then learn a scoring function to reverse this degradation and reconstruct the clean speech. The traditional forward process can be represented by the following stochastic differential equation (SDE):

[0049] da t = f(a,t)dt+g(t)dw,

[0050] In the formula, f is the drift coefficient, controlling the transformation of the mean during the diffusion process; g is the diffusion coefficient, controlling the amount of Gaussian white noise injected at each time step; and w is the standard Wiener process. The reverse process, i.e., solving this stochastic differential equation, can be expressed as:

[0051]

[0052] The scoring function at the current time step Typically, a neural network μ is parameterized. θ To make predictions.

[0053] A special property of the forward process of the above stochastic differential equation is that it has a closed-form solution at any time step t:

[0054] a t =α t a0+σ t ε,

[0055] In the formula, a t This represents all intermediate time points in the diffusion process, starting from clean data, and also represents the speech signal that is gradually contaminated by noise during the diffusion process. α t β represents the scaling factor for the clean signal a0.t The noise scaling function controls the ratio between the clean signal a0 and Gaussian noise during the noise addition process, and also represents the intensity of the noise injection. t represents the diffusion time step, indicating the stage of the current data in the entire noise addition or denoising process.

[0056] In this invention, the mean interpolation method is a special form of the standard process described above. By weighting the clean speech and the noisy signal, it allows the model to better utilize information when processing the data. This strategy is particularly suitable for the training process of diffusion models because it helps to smooth the transition and reduce the impact of noise on model performance. Specifically, the forward stochastic differential equation can be rewritten as:

[0057]

[0058] In this equation, Let ηt represent the perturbation term, and ηt be a time-dependent function representing the perturbation kernel of the SDE. According to the above equation, the noisy sample at any time step t is:

[0059] a t =α t [η t a o +(1-η t )c]+β t ε.

[0060] By comparing the above formulas, it can be seen that mean interpolation incorporates the noise signal into the forward SDE by interpolating a and c. Mean interpolation can be studied from the perspective of distribution enhancement. The above formulas can be expressed in a simpler form:

[0061]

[0062] Among them κ t This is a time-varying coefficient. The equation is similar in form to distribution augmentation, where speech noise n, Gaussian noise ε, and time step t all participate in the transformation process. Unlike traditional data augmentation methods that apply simple transformations to only a single sample, distribution augmentation introduces a series of transformations, enabling the model to learn the underlying distribution of the transformed data under specific conditions. This approach promotes better generalization ability by training the model on individual variations and their broader distribution.

[0063] Within a unified mean interpolation framework, this invention further proposes a distribution enhancement method that generates more diverse training samples by decoupling clean speech from its corresponding noise and performing shuffling and recombination. Specifically, firstly, the noise portion extracted from the original noisy speech signal (i.e., subtracting the clean speech from the noisy speech) is isolated to obtain pure noise signals unrelated to the speech content. Then, these noise samples are randomly shuffled, breaking their fixed correspondence with the original speech. Next, the shuffled noise is recombinated with multiple different clean speech signals to generate a new batch of noisy speech data. This process does not rely on the original speech-noise pairing, significantly expanding the coverage of the training samples. Compared to traditional methods, this scheme improves the diversity of training data and the model's adaptability to complex noise without introducing additional noise sources, demonstrating good practicality and generalizability. The forward process of this scheme can be described as follows:

[0064]

[0065] in Corresponding to 'n' in the previous formula, represents the decoupled and perturbed speech noise. This scheme is implemented during the training phase and maintains consistency with the original scheme during the inference phase. Therefore, this invention can generate samples through a standard inference process, as described above.

[0066] The speech predictor used in this embodiment is as follows: Figure 2 As shown, its input and output design is as follows: The speech predictor is based on a multi-resolution U-Net architecture, which performs well in generation and segmentation tasks. The network handles complex inputs by treating the real and imaginary parts of complex numbers as two separate channels to accommodate the network's requirements for processing real numbers. The network's input layer uses 3x3 Conv2D layers with a stride of 1. Feature extraction is performed in a shrinking path using residual blocks, each containing a Conv2D layer, group normalization, upsampling or downsampling of a Finite Impulse Response (FIR) filter, and a Swish activation function. Each upsampling layer consists of three residual blocks, and each downsampling layer consists of two residual blocks concatenated, with the last block performing either upsampling or downsampling. A global attention mechanism is introduced at a 16x16 resolution and in the bottleneck layer to better learn global dependencies in the feature maps.

[0067] The speech predictor network receives a complex spectrogram containing the speech signal, with the real and imaginary parts as inputs to different channels. The network outputs a complex spectrogram of the generated, clear speech, containing the corresponding real and imaginary parts. To make the model temporally dependent, the network architecture incorporates information about the progress of the current diffusion process. A common approach is to use Fourier embedding, mapping scalar time coordinates to an M-dimensional vector and integrating it into each residual block. The network also employs a progressively enhanced input mechanism, providing a downsampled version to each feature map in the contraction path, a method that performs well in high-resolution image generation. The downsampling operation shares weights for each resolution in the progressive enhancement, and the output in the expansion path follows the same progressive enhancement method. This design allows the network to efficiently generate high-quality speech signals while maintaining its ability to handle complex spectrograms.

[0068] Based on the aforementioned speech predictor, this embodiment provides a speech enhancement method based on a distributed enhancement diffusion model, the steps of which are as follows:

[0069] (1) Obtain the time embedding according to the set time step;

[0070] (2) Obtain the noisy speech embedding;

[0071] (3) Obtain the enhanced speech through a speech predictor;

[0072] The effectiveness of the distributed enhancement-based diffusion model speech enhancement method provided by this invention will be verified using a specific dataset.

[0073] (I) Dataset

[0074] This embodiment uses the VoiceBank-DEMAND dataset for model training and testing. This dataset contains known pairs of speech to be enhanced and clean speech, and it has been divided into training and testing sets. See reference [Botinhao, Cassia Valentini, et al. Investigating RNN-based speech enhancement methods for noise-robust text-to-speech. In 9th ISCA speechsynthesis workshop, 2016].

[0075] (II) Speech Predictor

[0076] This embodiment uses a deep neural network (Noise Conditional Fractional Network, NCSN++) as the speech predictor. For specific implementation, refer to the following reference: [J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speechenhancement and dereverberation with diffusion-based generative models,” TASLP, vol.31, pp.2351-2364, 2023].

[0077] (III) Indicators

[0078] To comprehensively evaluate the effectiveness of speech enhancement algorithms, this invention uses perceptual evaluation of speech quality (PESQ), short-term objective intelligibility (STOI), mean opinion score prediction of speech signal distortion (CSIG), mean opinion score prediction of background noise (CBAK), and mean opinion score prediction of overall effect (COVL). For each indicator, a higher score indicates better performance.

[0079] PESQ (Perceptual Speech Quality Assessment): An objective method for measuring speech quality that simulates the human ear's perception of audio quality by scoring the difference between a reference speech and the processed speech.

[0080] ESTOI (Extended Short-Time Objective Intelligibility): Used to assess the intelligibility of speech in noisy environments. It is calculated based on the correlation between reference speech and target speech in the short-time spectrum and is an important indicator of speech intelligibility.

[0081] CSIG (Mean Opinion Score Prediction of Speech Signal Distortion): Used to reflect the overall fidelity of enhanced speech, focusing on assessing the degree of signal distortion and reflecting the naturalness and clarity of speech.

[0082] CBAK (Mean Opinion Score Prediction for Background Noise): This is used to measure the degree of residual background noise in speech. The score reflects the cleanliness of the speech and the effectiveness of background noise suppression.

[0083] COVL (Mean Opinion Score Prediction of Overall Performance): Provides a comprehensive assessment of perceived audio quality, taking into account the effects of speech intelligibility and background noise.

[0084] (iv) Explanation of the remaining methods in the table

[0085] The parentheses following each method name represent two commonly used variance types for SDEs, where E represents Exploding (the noise scale β mentioned earlier). tIt grows explosively over time, while the scaling scale α of the clean signal t (Unchanged), P represents Preserving (the noise scale β mentioned above). t It increases slightly over time and scales with the clean signal by a scale α. t Keep the sum of squares equal to 1).

[0086] *: Metrics reflected by the test set data without any processing.

[0087] DIS: Discriminative speech enhancement model. For a detailed implementation, please refer to the following literature: [J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based generative models,” TASLP, vol. 31, pp. 2351-2364, 2023].

[0088] DM(E): A diffusion-based speech enhancement model without any additional design, using SDE with exploding variance.

[0089] DM(P): A diffusion-based speech enhancement model without any additional design, using variance-preserving SDE.

[0090] MIDM(E): A diffusion-based speech enhancement model that introduces mean interpolation and uses SDE with exploding variance.

[0091] MIDM(P): A diffusion-based speech enhancement model that introduces mean interpolation and uses variance-preserving SDE.

[0092] Ours: The speech enhancement diffusion model based on distribution enhancement proposed in this invention adopts a decoupled noise shuffling strategy within the mean interpolation framework.

[0093] The evaluation metrics obtained after evaluating all methods on the VoiceBank-DEMAND dataset are shown in Table 1. The best result for each metric is shown in bold.

[0094] Table 1

[0095]

[0096] Results Analysis: Table 1 reports the speech enhancement performance of the present invention and all baselines on the VoiceBank-DEMAND dataset. As shown in Table 1, the present invention achieves the best performance on all metrics of the VoiceBank-DEMAND dataset. The present invention and MIDM(P) both outperform DM(P) (without any noise enhancement), and MIDM(E) outperforms DM(E), highlighting that the interpolation method is essentially a data enhancement technique using paired noise signals. Introducing unpaired noise signals further improves performance, confirming the positive impact of the distributed enhancement method proposed in this invention.

[0097] In summary, this invention systematically rethinks and improves the mean interpolation strategy in existing diffusion models, reinterpreting it as a specialized distribution-enhanced diffusion kernel, thus providing new theoretical support and practical pathways for speech enhancement tasks. By unifying the interpolation method in the perturbation kernel of stochastic differential equations (SDEs), this invention demonstrates that mean interpolation is not merely a simple weighted interpolation operation, but a distribution enhancement method that actively introduces noise sample diversity and expands the distribution of training data through distribution transformation. Based on this framework, this invention further designs and introduces a decoupled noise shuffling mechanism—that is, recombining clean speech with unpaired noise to generate new speech-noise pairs, significantly expanding the coverage of the distribution seen by the model during the training phase. This shuffling method breaks the traditional one-to-one correspondence between noise and speech, enabling the model to learn to remain robust in more diverse and complex noise backgrounds, thereby effectively improving its adaptability to noisy scenarios and speech enhancement performance.

[0098] At the implementation level, this invention constructs a generalized SDE perturbation kernel that covers various existing mean interpolation variants by introducing unified interpolation coefficients into both standard variance explosion and variance preservation diffusion kernels. This innovation not only preserves the model's accurate simulation of the clean speech-noise mixture mechanism but also provides a flexible and controllable parameterization path for noise reconstruction. Experimental results show that the diffusion model applying the method of this invention reaches or exceeds the current state-of-the-art levels in several mainstream speech quality and intelligibility metrics, including PESQ, STOI, CSIG, CBAK, and COVL.

[0099] Furthermore, the method of this invention also possesses significant advantages in practical application. Since noise shuffling only occurs during the training phase, and the inference phase still utilizes the standard inverse SDE generation process, no additional computational overhead or complex noise manipulation is required. This means that it can be seamlessly integrated with existing diffusion model accelerated sampling techniques during deployment, ensuring the feasibility of real-time or near-real-time speech enhancement. In summary, this invention, by elevating mean interpolation to a diffusion kernel at the distribution enhancement level and combining it with a decoupled noise shuffling strategy, not only theoretically constructs a unified and comprehensive interpolation framework but also significantly improves the model's speech recovery capability in complex noise environments in practice, providing a practical and innovative solution for various real-world application scenarios—including remote communication, hearing aids, and intelligent human-computer interaction systems.

[0100] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. The scope of patent protection of the present invention shall be determined by the claims. Similarly, any equivalent structural changes made based on the content of the present invention's specification shall also be included within the scope of protection of the present invention.

Claims

1. A speech enhancement method based on a distributed enhancement diffusion model, characterized in that: Includes the following steps: S1. Constructing a noisy speech signal With clean speech signals and noise Relationship: ; S2. Based on the diffusion model, a forward process is used to gradually move towards clean speech signals. Add Gaussian noise to generate a noisy speech signal Its forward process can be represented by a generalized stochastic differential equation: in, The drift coefficient, The diffusion coefficient is... For standard Wiener process; S3. In the reverse process, the scoring function is learned through a neural network. To reconstruct a clean speech signal; A mean interpolation strategy is used to transform noisy speech signals. With clean speech signals The forward generalized stochastic differential equation for the weighted average is expressed as: in, Time-dependent perturbation kernel coefficients For disturbance terms; Mean interpolation strategy at any time step The noisy sample is represented as: ； Extending mean interpolation from the perspective of distribution enhancement will reduce noise. With Gaussian noise Similarly, the conversion process is introduced, and its expression is: in, The coefficient is a factor that varies with time. Further, it includes distribution enhancement steps: Decoupling noise from noisy speech signals A pure noise signal is obtained; Randomly shuffle the noise signal to generate perturbed noise. ; The noise after mixing Clean voice signal Recombined to generate new noisy speech samples; The forward process of distribution enhancement is represented as follows: in, This is the noise signal after rinsing; The method is applicable to single-channel speech enhancement tasks, used to process noisy speech input. Restore clean voice signal .

2. The speech enhancement method based on a distributed enhancement diffusion model according to claim 1, characterized in that: The forward process at any time step The closed solution is: in, Clean signal The scaling factor, For noise scaling function, Gaussian noise.

3. The speech enhancement method based on a distributed enhancement diffusion model according to claim 1, characterized in that: The generalized stochastic differential equation for the reverse process is expressed as: in, By neural network predict.

4. The speech enhancement method based on a distributed enhancement diffusion model according to claim 1, characterized in that: The distributed augmentation method is implemented during the training phase, and the speech augmentation results are generated by following the reverse process of the standard diffusion model during the inference phase.

Images