Industrial noise scene speech enhancement method and system based on rapid sampling denoising diffusion model

Abstract

The invention belongs to the technical field of data processing, and discloses an industrial noise scene speech enhancement method and system based on a fast sampling denoising diffusion model, and the method comprises the steps: carrying out the condition prior sampling of an input noisy speech through an accelerated sampling mechanism based on an information prior mechanism; according to a prior sampling result, performing a plurality of hop sampling through an accelerated sampling mechanism based on a diffusion model to obtain enhanced clean voice; the accelerated sampling mechanism based on the diffusion model carries out leapfrogging sampling on noisy voice through an ordinary differential equation described in a regularization mode. According to the method, two key mechanisms, leapfrog sampling and information prior-based sampling are utilized, so that the traditional speech enhancement efficiency and quality based on the de-noising diffusion probability model are improved, the total number of sampling steps is reduced, and the sampling process is remarkably accelerated, thereby reducing the consumption of computing resources, shortening the reasoning time and improving the speech enhancement efficiency and quality. The method is especially suitable for application scenes requiring rapid result generation.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and relates to a speech enhancement technology, and in particular to a speech enhancement method and system for industrial noise scenes based on a fast sampling denoising diffusion model. Background Art

[0002] Speech enhancement technology in industrial noise environments focuses on improving the clarity and quality of speech in the presence of non-stationary additive noise. This technology has important practical value in many industrial fields such as manufacturing, transportation, and energy industries. It can enhance the quality of voice communications, improve the effectiveness of voice control systems, and improve the accuracy and efficiency of automatic speech recognition systems. Given that industrial environments are more unique and complex than traditional environments, and that industrial environments place higher demands on the quality and processing speed of speech enhancement, achieving efficient and high-quality speech enhancement in industrial environments has become a challenging task.

[0003] At present, deep learning-based speech enhancement methods are mainly divided into two categories: discriminative methods and generative methods. Although discriminative models have long dominated the field of speech enhancement, generative models such as the Denoising Diffusion Probabilistic Model (DDPM) have also shown competitive performance and robustness in speech enhancement. However, in industrial noise environments, speech enhancement methods based on diffusion models still face many challenges. On the one hand, these methods usually require hundreds of inference steps to be evaluated through large networks, and their reverse process involves step-by-step sampling, resulting in a slow sampling rate. Although reducing the number of inference steps can reduce time complexity, this usually sacrifices the quality of the generated speech, making it difficult to find a balance between speed and performance. On the other hand, speech enhancement methods based on diffusion models also face the problem of a lack of relevant speech datasets, especially the lack of clean speech data paired with corresponding industrial noise, which further limits the application of these methods in actual industrial scenarios.

[0004] In summary, existing speech enhancement technologies based on denoising diffusion probability models generally have a large time cost problem because they require multiple iterative time steps in the inference process to synthesize high-quality speech samples. This high computational overhead directly hinders the deployment of denoising diffusion probability models in practical applications. Summary of the invention

[0005] The purpose of the present invention is to address the above technical problems existing in the prior art, propose a method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model, and introduce an accelerated sampler to sample faster and more efficiently.

[0006] The idea of ​​the present invention is to propose a sampler that can operate through two mechanisms: step-skipping sampling and sampling based on information prior. The first acceleration mechanism is that by re-parameterizing the differential equations and incorporating ordinary differential equations with more accurate Taylor expansion solutions, the sampler can safely skip steps and reduce the overall number of sampling times while maintaining the quality of speech enhancement. The second acceleration mechanism integrates information priors, allowing the reverse process to start with fewer steps, thereby further speeding up the process while maintaining high-quality output.

[0007] In real-world environments, speech is often interfered with by noise, where the noisy signal N represents a mixture of clean speech x and noise m. This relationship can be expressed by the following mathematical formula:

[0008] N=x+m

[0009] The goal of the speech enhancement (SE) task is to recover a clean speech signal x from a noisy input N. Diffusion-based speech enhancement models generate clean speech by learning the probabilistic relationship between noisy and clean speech. The process involves a forward and a backward phase. In the forward process, the data is diffused over time, and its dynamics are modeled by a stochastic differential equation (abbreviated as SDE) as follows:

[0010]

[0011] Where t represents the time step, x t represents the signal at time t, which represents the speech signal that is gradually contaminated by noise during the diffusion process. g(t) represents the drift function, which controls the signal x during the diffusion process. t The dynamic change trend of . h(t) represents the diffusion coefficient, which controls the intensity of noise injection. ψ t represents the standard Wiener process, which is used to generate random noise. x0~q0(x0) represents the probability distribution of the initial signal. The above forward process has an equivalent reverse process, defined as a diffusion model, which aims to reverse the diffusion and produce enhanced speech. The SDE description of the reverse process is as follows:

[0012]

[0013] Among them, q t (x t ) indicates that the signal x in the reverse process t The marginal probability distribution of . represents the key scoring function, which is composed of the neural network ∈ θ (x t ,t) calculation, indicating the signal x t The gradient of the log probability density function at time t. represents the standard Wiener process in inverse time, similar to ψ in forward diffusion t , but re-simulates the noise of the diffusion trajectory in the reverse process. Since the model is a conditional model, it uses both noisy and clean speech inputs and is optimized by the following loss function:

[0014]

[0015] Among them, ∈ represents Gaussian noise, ∈ θ (x t ,t,z) represents the noise predicted by the neural network, which estimates the noise component contained in the data during the diffusion process. z represents the noisy input signal (i.e. noisy speech).

[0016] Through the training of the forward and reverse phases, a diffusion model that meets the requirements is obtained (i.e. Figure 1 ). Figure 1 As shown in the figure, at each time step, the output of the previous time step, the embedding of time step t, and the conditional input are sent to the reverse module to obtain the output of this time step. Therefore, if there are T sampling steps in total, the noisy speech will pass through T reverse modules (iteration T times) during the processing process to obtain the final clean speech.

[0017] The first acceleration mechanism:

[0018] In the sampling process of DDPM, generating data requires discretizing the SDE and determining the appropriate number of sampling steps. Since reducing the number of steps can reduce the sampling time, an intuitive acceleration idea is to skip some of the original total time steps, such as reducing from 100 steps to 50 steps, instead of proceeding step by step. However, experiments have shown that using too few steps will introduce significant numerical errors, resulting in a decrease in the quality of the generated data. In order to alleviate these problems, the present invention uses ordinary differential equations (ODE for short) instead of SDE in the reverse process. ODE provides a deterministic alternative that reduces the randomness associated with the random process, thereby reducing errors in the reverse process. This allows the sampler to safely skip some steps without losing sampling quality. To achieve this, the key is to remove the random terms. While keeping x t Therefore, according to the SDE of the above reverse process, the corresponding ODE is:

[0019]

[0020] Going one step further, by replacing the scoring function with a noise predictor, the following probability flow ODE is defined:

[0021]

[0022] Among them, μ t Represents the scale of the noise at time t. It means that the signal at time T at the end of the forward diffusion process has been almost completely contaminated by noise, where Represents the variance of the noise data at the beginning of the reverse process, which can be set in advance as a hyperparameter; I represents the unit matrix. Compared with SDE, probability flow ODE has stronger determinism and supports fewer steps. The present invention uses an efficient ODE solver to achieve step jumping, thereby accelerating the sampling process. However, traditional ODE solvers are often regarded as "black boxes", and their ability to utilize specific model characteristics is greatly limited. To overcome this shackles, the present invention designs a "white box" solver for DDPMs derivation. For simplicity Formula, set is k(r,t), which is used to control the change of the signal from time step r to the current time step t. Then the ODE of the white box solver is expressed as:

[0023]

[0024] Here, the present invention provides a set of regularized equations to specifically describe the ODE of the reverse process, and these functions are expressed as:

[0025]

[0026] With these regularization equations, we set By parameterizing the above ODE, we can get the exact solution:

[0027]

[0028] Among them, x t and x D represents the signal at different time steps t and l. This equation consists of two parts. The first can be calculated exactly, and the second term Represents the cumulative effect of noise, which can be approximated by Taylor expansion and needs to be calculated by a noise predictor (such as a U-net with a residual module, a U-net with time encoding, or a deep neural network with a Transformer architecture):

[0029]

[0030] in, Represents an infinitesimal quantity.

[0031] This formula uses the intermediate time steps before the current time step η The estimated value of is close to the noise estimate of the current time step, where is in the middle time step The nth derivative on it. By adjusting the order k of the expansion, we can balance between the number of steps and the computational cost. A higher expansion order reduces the total number of steps but increases the computational complexity of each step. For industrial speech enhancement tasks, the present invention adopts a setting of k = 1, maintaining high-quality results by skipping certain steps while minimizing the total number of steps.

[0032] The noise predictor is a U-shaped network with a residual module, a U-shaped network with time encoding, or a deep neural network with a Transfomer architecture, etc. In a specific implementation manner, the noise predictor is one of Diffusion Transformer, DiffWave, Latent Diffusion, etc.

[0033] Based on the above analysis, the industrial noise scenario speech enhancement method provided by the present invention obtains enhanced clean speech by performing skip sampling on the input noisy speech several times through an accelerated sampling mechanism based on a diffusion model; the accelerated sampling mechanism based on the diffusion model performs skip sampling on the noisy speech through an ordinary differential equation described by regularization.

[0034] The present invention also provides an industrial noise scenario speech enhancement system based on a fast sampling denoising diffusion model, which includes a sampler, and the sampler is provided with a noise predictor. The sampler performs skip sampling on the input speech according to the accelerated sampling mechanism based on the diffusion model given above.

[0035] The second acceleration mechanism:

[0036] Although the first acceleration mechanism allows the reverse process to skip certain steps, the total length of the diffusion schedule does not change. Information prior is a concept in Bayesian statistics. In the Bayesian statistical framework, the prior refers to the assumption or belief about the probability distribution of the model parameters before observing any data. We found that introducing information prior can reduce the computational burden by starting sampling from a time step later in the time series. Based on this, the present invention starts the reverse denoising process from an intermediate time point τ < T to shorten the entire scheduling process and improve efficiency. Specifically, the standard forward process {x t} t∈[0,T] starts from x0 and evolves step by step according to the Gaussian distribution:

[0037] q 0t (x t |x0) = N(x t |x0α(t), μ 2 (t)I)

[0038] Where α(t) and μ(t) are continuous functions of time t, indicating how the clean signal x0 changes as noise is added. For the standard diffusion process, the reverse process can be expressed as:

[0039]

[0040] After introducing information prior, the above process can be adjusted as follows:

[0041]

[0042] Where x0 represents the enhanced clean speech; p θ (x t-X |x t ,z) represents the reverse sampling probability from time step t to t-1 under given condition z in the diffusion model, which is obtained by skipping sampling through the accelerated sampling mechanism based on the diffusion model; T represents the total time step; p(x H |z) is the conditional prior at time step τ, which can be achieved by the above standard forward diffusion:

[0043]

[0044] In the formula, z represents noisy speech, which serves as conditional information; represents the clean speech data without noise pollution at the beginning of the forward process; σ(τ) represents the noise intensity added at τ time steps; ∈ represents the noise term sampled from the standard normal distribution.

[0045] This approach significantly reduces the computational overhead (from T time steps to Tm time steps), speeds up the sampling of the reverse process, and has little impact on the denoising performance.

[0046] Based on the above analysis, the present invention also provides a method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model, which is characterized by comprising the following steps:

[0047] S1 performs conditional prior sampling on the input noisy speech through an accelerated sampling mechanism based on the information prior mechanism;

[0048] S2 performs several skip samplings based on the prior sampling results through an accelerated sampling mechanism based on a diffusion model to obtain enhanced clean speech; the accelerated sampling mechanism based on the diffusion model performs skip sampling on noisy speech through an ordinary differential equation described by regularization.

[0049] The present invention also provides an industrial noise scene speech enhancement system based on a fast sampling denoising diffusion model, comprising a first sampler and a second sampler; the second sampler is provided with a noise predictor;

[0050] The first sampler performs conditional a priori sampling on the input speech according to the accelerated sampling mechanism based on the information a priori mechanism given in the previous method;

[0051] The second sampler performs skip sampling on the input speech according to the accelerated sampling mechanism based on the diffusion model given in the previous method.

[0052] Since the speech enhancement method based on the diffusion model still faces the problem of lack of relevant speech data sets, the present invention designs and synthesizes a data set with industrial scene noise to simulate the real-world situation, alleviating the problem of data set scarcity in industrial speech enhancement. For clean speech, we used a well-known data set: the THCHS-30 data set. This speech data set contains 11,388 speech data from 40 speakers, with a total length of about 33.5 hours. The noise data set uses the ESC-50 data set. This data set consists of 5-second recordings and is divided into 5 main categories, each of which contains 10 semantic categories, for a total of 50 semantic categories. Each semantic category contains 40 examples. The present invention uses noise of ten speech categories under the main category of urban noises.

[0053] To generate training data, THCHS-30 speech samples and noise are randomly mixed in the training set, and the signal-to-noise ratio (SNR) is randomly selected from -5dB and 5dB. 50,000 training data are generated by combining the THCHS-30 training set with noise extracted from the random noise class. To generate validation data, the present invention creates 10 noise segments by mixing 10 clean samples with random noise classes in the same SNR range in the THCHS-30 validation set. For testing, 2495 clean samples in the test set are mixed with noise from 10 different noise categories to generate 7485 noise test samples at three signal-to-noise ratio levels (-5dB, 0dB, 5dB) to evaluate the performance of the model. This dataset containing industrial scene noise alleviates the problem of dataset scarcity in industrial speech enhancement. This dataset fills the gap of insufficient data in existing research to a certain extent, and also enhances the robustness and generalization ability of the algorithm in complex industrial environments by including multiple types of industrial noise. Furthermore, this dataset can serve as a standardized evaluation benchmark, facilitating fair comparisons among different speech enhancement methods and accelerating technical progress.

[0054] Compared with the prior art, the industrial noise scene speech enhancement method based on the fast sampling denoising diffusion model provided by the present invention has the following beneficial effects:

[0055] (1) The method for speech enhancement in industrial noise scenes based on the fast sampling denoising diffusion model proposes an accelerated sampler, which utilizes two key mechanisms: skip sampling and sampling based on information prior, which improves the efficiency and quality of traditional speech enhancement based on the denoising diffusion probability model, reduces the total number of sampling steps, and significantly speeds up the sampling process, thereby reducing the consumption of computing resources and shortening the inference time. It is particularly suitable for application scenarios that require fast generation of results.

[0056] (2) Compared with the traditional fixed reverse process, the industrial noise scene speech enhancement method based on the fast sampling denoising diffusion model of the present invention has greater flexibility and can dynamically adjust the sampling process according to the specific application scenario to adapt to various task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of an industrial noisy speech enhancement task, which describes the noisy speech processing flow chart without the accelerated sampling mechanism.

[0058] Figure 2 A schematic diagram of the principle of a method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model provided in Example 1 of the present invention.

[0059] Explanation of terms

[0060] DDPM is the abbreviation of Denoising Diffusion Probabiblistic Model, which means "denoising diffusion probability model", that is, a learning framework for generative models; its implementation method refers to [Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. In NIPS, 2020] DETAILED DESCRIPTION

[0061] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0062] Example 1

[0063] This embodiment provides a method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model. Taking speech enhancement processing including T time steps as an example, Figure 2 As shown in the figure, firstly, the accelerated sampling mechanism based on the information prior mechanism saves m time steps and directly starts the reverse process from the Tm-th time step; secondly, the accelerated sampling mechanism based on the diffusion model obtains the enhanced clean speech through multiple skip samplings, and the time steps skipped by each skip sampling can be the same or different.

[0064] In a specific implementation, the method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model provided in this embodiment includes the following steps:

[0065] S1 performs conditional prior sampling on the input noisy speech through an accelerated sampling mechanism based on an information prior mechanism.

[0066] The reverse process of the diffusion process of the accelerated sampling mechanism based on the information prior mechanism is expressed as:

[0067]

[0068] Where z represents noisy speech; p(x H |z) represents the conditional prior at time step τ under the given noisy speech condition; p θ (x t-X |x t ,z) represents the reverse sampling probability from time step t to t-1 under given condition z in the diffusion model, which is obtained by skipping sampling through the accelerated sampling mechanism based on the diffusion model; T represents the total time step;

[0069] Among them, p(x H |z) is obtained by standard forward diffusion:

[0070]

[0071] represents the clean speech used in the forward training process; σ(τ) represents the noise intensity added at τ time steps; ∈ represents the noise term sampled from the standard normal distribution.

[0072] like Figure 2 As shown, the physical examination prior p(x H |z), thus omitting the first m time steps, that is, τ = Tm.

[0073] S2 performs several skip samplings based on the prior sampling results through an accelerated sampling mechanism based on a diffusion model to obtain enhanced clean speech; the accelerated sampling mechanism based on the diffusion model performs skip sampling on noisy speech through an ordinary differential equation described by regularization.

[0074] The ordinary differential equation described by regularization is:

[0075]

[0076] In the formula, x t and x D represents the signal at different time steps t and l; α represents a continuous function; represents the prediction noise; μ represents the scale of the noise.

[0077] By Taylor expansion:

[0078]

[0079] Where k represents the order of expansion (k = 1); The noise predictor is predicted by a noise predictor. The noise predictor is a U-type network with a residual module, a U-type network with time coding, or a deep neural network of a Transfomer architecture. In a specific implementation, the noise predictor is one of Diffusion Transformer, DiffWave, LatentDiffusion, etc.

[0080] like Figure 2 As shown, through the accelerated sampling mechanism based on the diffusion model, n1 time steps or n2 time steps can be skipped each time for sampling.

[0081] This embodiment also provides an industrial noise scene speech enhancement system based on a fast sampling denoising diffusion model, comprising a first sampler and a second sampler; the second sampler is provided with a noise predictor;

[0082] The first sampler performs conditional a priori sampling on the input speech according to the accelerated sampling mechanism based on the information a priori mechanism given by the above method;

[0083] The second sampler performs skip sampling on the input speech according to the accelerated sampling mechanism based on the diffusion model given in the above method.

[0084] System training process: First, the training data is used to train the above system through the forward process and the reverse process. The loss function Loss given above is used in the training process. The noise predictor parameters are optimized through the gradient descent optimization algorithm until the loss function tends to be unchanged, that is, the system training is completed.

[0085] Example 2

[0086] When the noisy speech to be processed is not very complex, the input noisy speech can be directly subjected to several skip samplings through an accelerated sampling mechanism based on a diffusion model to obtain enhanced clean speech.

[0087] Therefore, this embodiment provides a speech enhancement method for industrial noise scenes based on a fast sampling denoising diffusion model, which performs skip sampling on the input noisy speech several times through an accelerated sampling mechanism based on a diffusion model to obtain enhanced clean speech; the accelerated sampling mechanism based on the diffusion model performs skip sampling on the noisy speech through an ordinary differential equation described by regularization.

[0088] The ordinary differential equation described by regularization is:

[0089]

[0090] In the formula, x t and x D represents the signal at different time steps t and l; α represents a continuous function; represents the prediction noise; μ represents the scale of the noise.

[0091] By Taylor expansion:

[0092]

[0093] Where k represents the order of expansion (k = 1); Predicted by the noise predictor.

[0094] The noise predictor is a U-type network with a residual module, a U-type network with time coding, or a deep neural network of a Transformer architecture, etc. In a specific implementation, the noise predictor is one of Diffusion Transformer, DiffWave, Latent Diffusion, etc.

[0095] This embodiment also provides an industrial noise scene speech enhancement system based on a fast sampling denoising diffusion model, including a sampler, and the sampler is provided with a noise predictor; the sampler performs skip sampling on the input speech according to the accelerated sampling mechanism based on the diffusion model given in the above method.

[0096] System training process: First, the training data is used to train the above system through the forward process and the reverse process. The loss function Loss given above is used in the training process. The noise predictor parameters are optimized through the gradient descent optimization algorithm until the loss function tends to be unchanged, that is, the system training is completed.

[0097] Application Examples

[0098] (I) Dataset

[0099] This embodiment uses the above-mentioned synthetic industrial noisy speech data set, in which the training set contains 50,000 noisy speech, the validation set contains 10 noisy speech, and the test set contains 7,485 noisy speech. For this data set, each noisy speech has a corresponding clean speech without noise.

[0100] (II) Noise Predictor

[0101] This embodiment uses a deep neural network (a non-autoregressive feedforward bidirectional atrous convolutional neural network, DiffWave, for specific implementation reference [Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, Bryan Catanzaro. Diffwave: Aversatilediffusion model for audio synthesis. In ICLR, 2020]) as a noise predictor.

[0102] Four baseline methods (DiffuSE, CDiffuSE, SGMSE, DR-DiffuSE) are applied to the test set of the above data set to perform speech enhancement tasks. Then, the industrial noise scene speech enhancement method based on the fast sampling denoising diffusion model provided in Example 1 (abbreviated as DISEN) is used for training in the training set and verified using the validation set to obtain a system for speech enhancement; then the system is used to process and evaluate the noisy speech in the test set. The number of times the deep neural network is called is used as a criterion for measuring time efficiency. In order to comprehensively evaluate the effectiveness of the speech enhancement algorithm, the present invention adopts the following four key evaluation indicators:

[0103] PESQ (Perceptual Speech Quality): This is a widely used objective metric for evaluating the perceptual quality of speech signals. The typical range of PESQ scores is from -0.5 to 4.5, where higher scores indicate better perceptual quality. This metric helps us understand whether the enhanced speech is closer to the natural speech quality perceived by the human ear.

[0104] STOI (Short Time Objective Intelligibility): This metric is used to quantify the intelligibility of speech, and is evaluated by measuring the similarity between the enhanced speech and the original clean speech. The STOI score ranges from 0 to 1, with higher scores indicating higher speech intelligibility. STOI is particularly useful for evaluating speech intelligibility in noisy environments.

[0105] CSIG: This indicator is used to measure the degree of distortion between the enhanced speech and the reference clean signal, reflecting the overall quality of the signal. The CSIG score range is from 1 to 5, and the higher the score, the better the signal quality and the lower the distortion.

[0106] CBAK: This indicator is used to evaluate the level of background noise remaining in the enhanced speech, thus reflecting the effectiveness of noise suppression. CBAK also uses a rating range of 1 to 5. The higher the score, the better the background noise is suppressed and the clearer the speech is.

[0107] The evaluation metrics obtained after testing all methods on the test set are shown in Table 1. The number of time steps of the reverse process (the number of times the noise predictor is called) is also used to evaluate the model efficiency. The best result at each decibel level is shown in bold, and the suboptimal result is underlined.

[0108] The remaining methods in the table are explained as follows:

[0109] DiffuSE: The full name is Diffusion probabilistic model-based SE. This model uses DDPM to complete the speech enhancement task. Its core method is to gradually remove the noise in the speech during the back diffusion process and restore the clean speech signal. When dealing with a more complex noise environment, DiffuSE improves the enhancement effect through successive iterations, especially when the signal-to-noise ratio is high. The implementation process can be found in the literature [Lu, Y.-J.; Tsao, Y.; and Watanabe, S. A. study on speech enhancement based on diffusion probabilistic model. In APSIPAASC, 2021]

[0110] CDiffuSE: The full name is Condition Diffusion probabilistic model-based SE. This method is a conditional version of the DiffuSE method. CDiffuSE optimizes the denoising effect by inputting the speech to be enhanced as conditional information into the diffusion process. Unlike DiffuSE, this model starts training from noisy speech instead of Gaussian noise, which enables it to guide the reverse diffusion process more efficiently, especially in complex noise scenarios. The implementation process references [Lu, Y.-J.; Wang, Z.-Q.; Watanabe, S.; Richard, A.; Yu, C.; and Tsao, Y. Conditional diffusion probabilistic model for speech enhancement. In ICASSP, 2022]

[0111] SGMSE: The full name is Speech Enhancement with Score-Based Generative Models, which is a speech enhancement method based on a score generative model. It uses a score generative model, which is a generalized form of the diffusion model. This method gradually removes noise by solving SDE and achieves speech enhancement in the complex short-time Fourier transform (STFT) domain. The innovation of SGMSE lies in its ability to handle different types of noise scenes, and has strong generalization ability in complex noise environments, and is suitable for dynamic and diverse noise environments. The implementation process references [Welker, S.; Richter, J.; and Gerkmann, T..Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain. In Interspeech, 2022]

[0112] DR-DiffuSE: The full name is Denoising Refinement Diffusion SE. This is an improved diffusion model for speech enhancement by enhancing conditional information. It solves the conditional collapse problem and optimizes the efficiency and quality of diffusion denoising by introducing fast sampling and refinement networks. DR-DiffuSE can not only generate higher quality speech, but also further fix the residual problems after denoising, so that the output speech performs better in low signal-to-noise ratio environments. Its implementation process references [Tai, W.; Zhou, F.; Trajcevski, G.; and Zhong, T. Revisiting Denoising Diffusion Probabilistic Models for Speech Enhancement: Condition Collapse, Efficiency and Refinement. In AAAI, 2023]

[0113] From the experimental results in Table 1, it can be seen that DISEN performs particularly well in industrial scenarios, and its speech quality and processing efficiency are comparable to the current most advanced diffusion models. Despite using fewer sampling steps, DISEN still provides high-quality speech enhancement with minimal impact on performance. In contrast, DR-DiffuSE is not designed specifically for industrial noise environments, and its optimization focus is more on speech enhancement in general environments. The present invention further explores performance optimization in complex industrial noise scenarios and has made more significant progress in acceleration. In DISEN, we introduced a skip sampling strategy that allows sampling to skip some time steps to greatly increase the sampling speed. In addition, by starting reverse diffusion from the middle time step, we reduce the maximum number of steps, further reducing the computational burden. These innovations enable DISEN to efficiently map noisy speech in industrial environments to clean speech, significantly reducing the sampling steps while maintaining excellent enhancement quality. Experimental results in industrial scenarios show that DISEN not only improves the efficiency of speech enhancement, but also outperforms existing methods such as DR-DiffuSE in actual industrial scenarios, proving its robustness and superiority in dealing with complex and changeable industrial noisy speech.

[0114] In summary, the present invention uses the denoising diffusion probability model (DDPM) for speech enhancement and adopts two acceleration mechanisms: skip sampling and reducing the maximum number of steps. First, through the skip strategy, some time steps are skipped, which greatly speeds up the sampling speed; second, by starting the reverse diffusion from the middle time step, the maximum number of steps is reduced, further reducing the computational burden. Combining these strategies, the present invention can efficiently achieve the goal of

[0115] The mapping of noisy speech to clean speech significantly reduces the sampling steps while ensuring the quality of speech enhancement, and uses the noise predictor to optimize the denoising process, thereby achieving speech enhancement efficiency that is superior to existing methods.

[0116] Table 1: Effect and time efficiency of speech enhancement of various methods on synthetic datasets

[0117]

[0118] *: Indicators reflected by the test set data without any processing

[0119] Step refers to the number of time steps sampled in the reverse process. More steps means a more refined sampling process, but it also increases the computational overhead and time cost. Reducing the sampling steps can speed up the generation process.

[0120] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model, characterized in that: The input noisy speech is subjected to several skip samplings through an accelerated sampling mechanism based on a diffusion model to obtain enhanced clean speech; the accelerated sampling mechanism based on the diffusion model performs skip sampling on the noisy speech through an ordinary differential equation described by regularization.

2. The method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model according to claim 1 is characterized in that: The ordinary differential equation described by regularization is: In the formula, x t and x l represents the signal at different time steps t and l; α represents a continuous function; represents the prediction noise; μ represents the scale of the noise.

3. The method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model according to claim 2 is characterized in that: By Taylor expansion: In the formula, k represents the order of expansion; Predicted by the noise predictor.

4. The method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model according to claim 3 is characterized in that: The noise predictor is a U-type network with a residual module, a U-type network with time coding, or a deep neural network with a Transformer architecture.

5. A speech enhancement system for industrial noise scenes based on a fast sampling denoising diffusion model, characterized in that: The method comprises a sampler provided with a noise predictor; the sampler performs skip sampling on the input speech according to the accelerated sampling mechanism based on the diffusion model given in the method according to any one of claims 1 to 4.

6. A method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model, characterized in that: The following steps are involved: S1 performs conditional prior sampling on the input noisy speech through an accelerated sampling mechanism based on the information prior mechanism; S2 performs several skip samplings based on the prior sampling results through an accelerated sampling mechanism based on a diffusion model to obtain enhanced clean speech; the accelerated sampling mechanism based on the diffusion model performs skip sampling on noisy speech through an ordinary differential equation described by regularization.

7. The method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model according to claim 6 is characterized in that: In step S1, the reverse process of the diffusion process of the accelerated sampling mechanism based on the information prior mechanism is expressed as: Where z represents noisy speech; p(x / |z) represents the conditional prior at time step τ under the given noisy speech condition; p θ (x t-1 |x t ,z) represents the reverse sampling probability from time step t to t-1 when the condition z is given in the diffusion model, which is obtained by skipping sampling through the accelerated sampling mechanism based on the diffusion model; Among them, p(x / |z) is obtained by standard forward diffusion: In the formula, z represents noisy speech; represents the clean speech data without noise pollution at the beginning of the forward training process; σ(τ) represents the noise intensity added at the τ time step; ∈ represents the noise term sampled from the standard normal distribution.

8. The method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model according to claim 6 is characterized in that: In step S2, the ordinary differential equation described by regularization is: In the formula, x t and x l represents the signal at different time steps t and l; α represents a continuous function; represents the prediction noise; μ represents the scale of the noise.

9. The method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model according to claim 8 is characterized in that: By Taylor expansion: In the formula, k represents the order of expansion; Predicted by the noise predictor.

10. The method for speech enhancement in industrial noise scenes based on a fast sampling denoising diffusion model according to claim 9, characterized in that: The noise predictor is a U-type network with a residual module, a U-type network with time coding, or a deep neural network with a Transformer architecture.

11. A speech enhancement system for industrial noise scenes based on a fast sampling denoising diffusion model, characterized in that: The device comprises a first sampler and a second sampler; the second sampler is provided with a noise predictor; The first sampler performs conditional a priori sampling on the input speech according to the accelerated sampling mechanism based on the information a priori mechanism provided in the method of claim 6 or 7; The second sampler performs skip sampling on the input speech according to the accelerated sampling mechanism based on the diffusion model provided in the method according to any one of claims 6 to 10.

Images