A single-step speech enhancement method and system based on a drift model

Abstract

The application discloses a single-step speech enhancement method and system based on a drift model, and belongs to the technical field of speech signal processing. The application moves the distribution evolution process from the inference stage to the training stage, constructs a drift field composed of attractive terms and repulsive terms in a multi-scale feature space to drive network parameter updating during training, so that the network directly learns the residual correction amount from the noisy spectrum to the clean spectrum; during inference, only a single forward propagation is required, and a modulation vector adaptively generated from the energy of the input signal is introduced as a condition to realize differentiated enhancement under different signal-to-noise ratios. Experiments show that the single-step inference of the present application can obtain a real-time factor of 0.016 on the VoiceBank dataset, the speed is about 4.8 times that of the existing flow matching method, and the enhancement quality is flat with that of the latter multi-step inference, and the present application is suitable for real-time speech enhancement scenes.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a single-step speech enhancement method and system based on a drift model. Background Technology

[0002] In applications such as real-time communication, hearing aids, and intelligent voice interaction, stringent requirements are placed on the real-time performance, stability, and enhancement quality of speech enhancement. Achieving high-quality speech enhancement under extremely low latency constraints has long been a core technological challenge in this field.

[0003] Currently, among generative model-based speech enhancement methods, Flow matching schemes have attracted widespread attention due to their deterministic mapping characteristics. Existing FlowSE technical solutions typically involve two phases: training and inference. During training, the network learns a deterministic ordinary differential equation velocity field, which defines the single-step evolution direction from noisy speech distribution to clean speech distribution. During inference, starting from the time-frequency domain representation of the noisy speech, a multi-step numerical integration is performed using an ordinary differential equation solver. Each step requires inputting the current state into the network to complete a full forward propagation to calculate the velocity field vector under that state. Through multiple iterations, the network gradually approximates the clean speech. This method, by default, requires multiple inference steps to achieve optimal enhancement quality.

[0004] However, the above-mentioned schemes have inherent technical flaws in practical applications. First, the inference phase requires multi-step iterative solution of ordinary differential equations, with each step requiring a complete network forward propagation, resulting in high end-to-end inference latency. Even reducing the number of integration steps cannot achieve single-step completion, making it difficult to meet the needs of applications with extreme real-time requirements. Second, the network learns a single-step velocity field during training, while the inference phase applies this velocity field through multi-step integration. The inconsistency between the training and inference distributions makes it difficult to guarantee the stability of the enhancement effect, and this difference cannot be directly eliminated through optimization during training. Furthermore, the multi-step iterative process suffers from error accumulation; the bias introduced by the previous integration step propagates to subsequent steps and is gradually amplified, limiting further improvement in enhancement quality. The root cause of these flaws is that existing schemes completely confine the distribution evolution process to the inference phase. The network itself only learns single-step evolution rules and does not internalize the global distribution information required for multi-step evolution into the network parameters, making single-step inference theoretically impossible, and inevitably leading to distribution bias between training and inference. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a single-step speech enhancement method and system based on a drift model. This invention shifts the distribution evolution process from the inference stage to the training stage. It utilizes a drift field constructed in a multi-scale feature space, consisting of attracting and repulsive terms, to drive the training of the drift field network. This allows the network to directly learn the residual correction from the compressed complex spectrum to the clean compressed spectrum. During inference, only a single forward propagation is needed to complete speech enhancement, reducing the computational cost to a fraction of that of traditional stream matching methods. Furthermore, both training and inference perform the same single forward propagation, eliminating the inconsistency between the training and inference distributions in multi-step integration methods. In addition, an energy-adaptive conditional signal generation mechanism generates a conditional signal based on the average amplitude of the compressed complex spectrum and maps it to a modulation vector, injecting it into each residual block of the drift field network. This enables the network to perceive the signal-to-noise ratio characteristics of the input signal and perform differentiated enhancement processing for inputs with different noise conditions. The drift field network employs a hybrid loss scheme jointly optimized by drift loss, reconstruction loss, and multi-resolution STFT loss. The training process does not require distillation of the pre-trained model and can be completed end-to-end, resulting in stable and flexible training. Single-step forward propagation eliminates the need for complex sampler design, avoids the error accumulation problem caused by multi-step iteration, and enhances the stability of the system.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: a single-step speech enhancement method based on a drift model, comprising: acquiring the compressed complex spectrum of a noisy speech signal; A conditional signal is generated based on the average amplitude of the compressed complex spectrum, and a modulation vector is generated based on the conditional signal. The compressed complex spectrum is input into a pre-trained drift field network. Through a single forward propagation of the drift field network, the modulation vector is used as a conditional input to obtain an enhanced spectrum estimate. The enhanced spectrum estimate is converted into a time-domain signal, and the enhanced speech is output. The training of the drift field network includes: constructing a drift field V in a multi-scale feature space for the feature vector of the currently output enhanced spectral estimation. (p,q) (x) is attracted by the term and exclusion term Composition, that is , where x is from the generating distribution q The feature vector of the enhanced sample; The attraction term calculation x points to a clean speech distribution. p Multiple positive sample features y + The weighting direction pulls x toward the clean speech distribution. p The attraction item for: Among them, y j + For the first j A positive sample feature, k Z is the kernel function. p (x) is the normalization factor; The exclusion term calculation x points to the generated distribution. q Other enhanced sample features y - The weighting direction pushes x away from the generated distribution. q The exclusion term for: Among them, y k - For the first k Each negative sample feature, Z q (x) is the normalization factor; The drift field is used to drive the parameter updates of the drift field network.

[0007] Preferably, the condition signal t Generate in the following way: Calculate the mean value E of the amplitude at each time-frequency position of the compressed complex spectrum: The conditional signal is obtained by mapping the mean E as follows. t : in, F Number of frequency channels; T Y(time frame number) f , u ) represents the compressed complex spectrum at frequency f and time u The value at; s `clamp` is the sigmoid function; `clamp` is the truncation function. x 0 is the numerical stability constant for logarithmic operations; x 1 is the cutoff constant.

[0008] Preferably, the step of generating a modulation vector based on the conditional signal... c ,include: For the condition signal t Taking the logarithm, we get t log =log( t ); right t log Perform Gaussian Fourier projection to obtain the projection feature e fourier : Where W is a fixed random weight vector; The projection feature e fourier Mapped to the modulation vector via a fully connected network c : In this context, Linear is a fully connected layer, and swish is an activation function.

[0009] Preferably, the drift field network adopts the NCSN++ architecture, which has an encoder-decoder structure, skip connections, and a self-attention mechanism.

[0010] Preferably, obtaining the enhanced spectrum estimate includes: The residual correction is output through a single forward propagation of the drift field network. O ; The residual correction amount O With the compressed complex spectrum Y Add them together to obtain the enhanced spectrum estimate. : .

[0011] Preferably, the training of the drift field network further includes: Integrating drift loss using a weighted summation method Reconstruction loss and multi-resolution STFT loss To obtain the total training loss in, oh d , oh r , oh s These are the weighting coefficients for each type of loss; Based on the total training loss L Update the parameters of the drift field network.

[0012] Preferably, the feature vectors of the multi-scale feature space are extracted through at least one of the following methods: Manual statistical features are used to extract statistics at multiple scales in the original spectral domain and / or the Mel spectral domain. Features extracted from a pre-trained self-supervised speech model.

[0013] A single-step speech enhancement system employing the above method includes: a time-frequency conversion module for converting noisy speech signals into compressed complex spectra; An energy adaptive condition module is used to generate a modulation vector based on the average amplitude of the compressed complex spectrum; A drift field network processing module is used to receive the compressed complex spectrum and the modulation vector, and obtain an enhanced spectrum estimate through a single forward propagation. A signal reconstruction module is used to convert the enhanced spectrum estimate into enhanced speech.

[0014] Preferably, the drift field network processing module adopts the NCSN++ architecture, which has an encoder-decoder structure, skip connections, and a self-attention mechanism.

[0015] Preferably, the following modules are also included for use during the training phase: The multi-scale feature extraction module is used to extract multi-scale feature vectors from the enhanced spectrum estimation and the clean compressed spectrum, respectively. The positive sample feature library module is used to accumulate multi-scale feature vectors of clean speech across batches. The drift field calculation module is used to calculate the drift field vector consisting of attraction and repulsion terms in the normalized feature space, based on the multi-scale feature vector and the features in the positive sample feature library. The hybrid loss module is used to integrate drift loss in a weighted summation manner. Reconstruction loss and multi-resolution STFT loss To obtain the total training loss L .

[0016] By adopting the above technical solution, the present invention has the following beneficial effects.

[0017] (1) This invention addresses the fundamental shortcomings of existing diffusion models and flow matching methods, which suffer from high latency and inability to meet real-time requirements due to multi-step iterative inference, by introducing a drift field-driven single-step inference mechanism. During the training phase, a drift field consisting of attracting and repulsive terms is constructed in the multi-scale feature space. The attracting term pulls the enhanced sample features toward the clean speech distribution, while the repulsive term pushes them away from their own generated distribution. Both terms jointly drive the parameter update of the drift field network, enabling the network to directly learn the residual correction from the noisy spectrum to the clean spectrum. The enhancement can be completed with only a single forward propagation during the inference phase, eliminating the error accumulation caused by multi-step integration in principle, and ensuring complete consistency between training and inference behavior. Experiments show that on the VoiceBank dataset, this invention can achieve a real-time factor of 0.016 with single-step inference, and the inference speed is approximately 4.8 times that of FlowSE. At the same time, the PESQ and ESOI indices are on par with the latter's 5-step inference.

[0018] (2) This invention utilizes an energy-adaptive conditional signal generation mechanism to map the global energy mean of the compressed complex spectrum into a conditional signal via a Sigmoid function. This signal is then transformed using a logarithmic transform, Gaussian-Fourier projection, and a fully connected network to generate a modulation vector, which is then injected into each residual block of the drift field network. This mechanism enables the network to perceive the signal-to-noise ratio characteristics of the input signal and perform differentiated residual corrections for strong and weak noise inputs, rather than employing a fixed processing strategy. The energy-adaptive conditional signal generation mechanism of this invention allows for the coverage of the entire scene's enhancement requirements with a single forward propagation, maintaining stable enhancement quality even under extremely low latency constraints.

[0019] (3) This invention extracts multi-scale statistical features from the original spectral domain and the Mel spectral domain respectively, forming a dual-domain complementary feature space for drift field calculation. The original spectral domain features retain phase information and fine frequency structure, ensuring signal fidelity; the Mel spectral domain features align with the human ear's critical frequency band perception characteristics, optimizing auditory quality. The drift field calculation uses a bidirectional normalized coupling matrix to ensure that each enhanced sample is guided towards a clean distribution, while each clean sample provides effective supervision. This dual-domain complementary mechanism, in conjunction with the attraction-repulsion structure of the drift field, enables distribution-level optimization to both approximate the target distribution as a whole and maintain the diversity of generated samples, thus achieving high-quality enhanced speech even under single-step output constraints.

[0020] (4) The drift field network of this invention adopts the NCSN++ architecture, and realizes the extraction and fusion of multi-resolution features through encoder-decoder structure, skip connections and self-attention mechanism. The modulation vector is injected into each residual block in the form of additive bias, realizing the fine control of the feature expression of each layer of the network by the conditional signal. With the residual learning strategy, the network output layer directly predicts the residual correction amount and adds it to the input to obtain the enhancement result. The output approaches zero in the initial stage of training, and learns to correct from the noisy input step by step. The training process converges stably. The adaptation of this architecture to the drift field training mechanism makes the complex distribution evolution process internalized in the network parameters, and the enhancement result can be directly output without any iterative solution during inference.

[0021] (5) This invention employs a hybrid loss scheme that weights and sums drift loss, reconstruction loss, and multi-resolution STFT loss. Drift loss applies regularization at the distribution level, pushing the overall generated distribution toward a clean distribution; reconstruction loss provides point-to-point fidelity supervision for each sample; and multi-resolution STFT loss provides complementary spectral convergence supervision in the waveform domain. Drift loss prevents pattern averaging and overfitting during training with pure reconstruction loss, reconstruction loss ensures single-sample accuracy, and STFT loss ensures auditory quality in the waveform domain. This hybrid loss scheme enables the network to achieve enhancement metrics comparable to multi-step methods under strict constraints of single-step output, and the training process does not rely on pre-trained model distillation, and can be completed end-to-end.

[0022] (6) This invention transfers the distribution evolution process from the inference stage to the training stage, and uses a drift field constructed in a multi-scale feature space, consisting of attracting and repulsive terms, to drive the training of the drift field network. This allows the network to directly learn the residual correction from the compressed complex spectrum to the clean compressed spectrum. During inference, only a single forward propagation is needed to complete the speech enhancement, reducing the computational load of inference to a fraction of that of traditional stream matching methods. Furthermore, both training and inference perform the same single forward propagation, eliminating the problem of inconsistent training and inference distributions in multi-step integration methods. In addition, through an energy-adaptive conditional signal generation mechanism, a conditional signal is generated based on the average amplitude of the compressed complex spectrum and mapped to a modulation vector, which is then injected into each residual block of the drift field network. This enables the network to perceive the signal-to-noise ratio characteristics of the input signal and perform differentiated enhancement processing for inputs with different noise conditions. The drift field network adopts a hybrid loss scheme jointly optimized by drift loss, reconstruction loss, and multi-resolution STFT loss. The training process does not require distillation of the pre-trained model and can be completed end-to-end, resulting in stable and flexible training. Single-step forward propagation eliminates the need for complex sampler design, avoids the error accumulation problem caused by multi-step iteration, and enhances the stability of the system. Attached Figure Description

[0023] The following provides a detailed discussion of the manufacture and application of preferred embodiments of the present invention. However, it should be understood that the present invention provides many applicable inventive concepts that can be embodied in various specific environments. The specific embodiments discussed are merely illustrative of specific ways of manufacturing and using the present invention and do not limit the scope of the invention. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0024] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0025] The following provides a detailed discussion of the manufacture and application of preferred embodiments of the present invention. However, it should be understood that the present invention provides many applicable inventive concepts that can be embodied in various specific environments. The specific embodiments discussed are merely illustrative of specific ways of manufacturing and using the invention and do not limit the scope of the invention.

[0026] This invention moves the distributed evolution process from the inference stage to the training stage. During training, a drift field composed of attracting and repulsive terms is constructed in a multi-scale feature space to drive network parameter updates, allowing the network to directly learn the residual correction from the noisy spectrum to the clean spectrum. During inference, only a single forward propagation is required, and a modulation vector adaptively generated from the input signal energy is introduced as a condition to achieve differentiated enhancement under different signal-to-noise ratios. Thus, high-quality enhanced speech signal pickup is achieved under the constraint of extremely low latency in single-step inference.

[0027] The following is in conjunction with the appendix Figure 1 To elaborate further.

[0028] like Figure 1 This paper presents a single-step speech enhancement method based on a drift model. This method moves the distributed evolution process from the inference stage to the training stage. It utilizes a drift field constructed in a multi-scale feature space, consisting of attracting and repulsive terms, to drive network parameter updates. This allows the drift field network to directly predict the residual correction from the compressed complex spectrum of the noisy speech in a single forward propagation. Combined with an energy-adaptive conditional signal, it achieves differentiated enhancement processing under different signal-to-noise ratios, thus obtaining high-quality enhanced speech under the extremely low latency constraint of single-step inference. The method specifically includes the following steps.

[0029] Step S1: Obtain the compressed complex spectrum of the noisy speech signal.

[0030] Specifically, first read the noisy speech signal y ( n Calculate the peak amplitude of the signal. c =max n | y ( n The waveform is then subjected to peak normalization to obtain a normalized signal. y norm ( n ): y norm ( n )= y ( n ) / c Normalization factor c Save the data for later inference to restore the original signal amplitude. Perform a short-time Fourier transform (STFT) on the normalized signal to convert the time-domain waveform into a complex spectrum representation in the time-frequency domain. Y stft ( f , u ): in, n This is the index of the sampling point within the current analysis window, i.e., the first sampling point within the current frame. n One sampling point; w ( n ) is the window function at the th n The values ​​at each sampling point are, in this embodiment, periodic Hann window functions; H Frame shift represents the number of sampling points between the start positions of two adjacent frames; f Frequency index; u For frame indexing; j Imaginary unit; Nfft This is the window length.

[0031] In this embodiment, the parameters of the short-time Fourier transform are set as follows: the number of FFT points is... N fft =510, corresponding to 256 frequency channels; frame shift H 128 sampling points were selected; a periodic Hann window was used as the window function, and a center-aligned mode was adopted to symmetrically fill both ends of the signal to obtain the complex spectrum. Y stft ( f , u After obtaining the complex spectrum, it is subjected to nonlinear power-law compression to obtain the compressed complex spectrum. Y ( f , t ):

[0032] Among them | Y stft | represents the amplitude spectrum, ∠ Y stft For phase spectrum, p The compression index (the square root of the amplitude is used to compress the dynamic range). α This is a scaling factor (scaling the numerical range to a range suitable for network training). This transformation preserves the original phase information and applies nonlinear compression only to the amplitude.

[0033] In this embodiment, the compression index is preferably 0.5, which means that the amplitude is compressed by the square root while retaining the original phase information completely, and a scaling factor is introduced to scale the numerical range to a range suitable for network training. During the training phase, the clean speech signal is processed in the same way as described above to obtain the corresponding clean compressed spectrum, which serves as the training target.

[0034] Step S2: Generate a conditional signal based on the average amplitude of the compressed complex spectrum, and generate a modulation vector based on the conditional signal.

[0035] In the energy feature extraction sub-step, the mean E of the amplitude at all time-frequency positions on the compressed complex spectrum is calculated. This mean E is calculated as follows: in, F To compress the number of channels in the frequency dimension of the complex spectrum; T To compress the number of frames in the time dimension of the complex spectrum; tThe conditional signal is calculated by summing the absolute values ​​of the amplitudes across all frequency channels and time frames in the compressed complex spectrum, and then dividing by the total number of points, which is the product of the number of frequency channels and the number of time frames. This mean value E directly reflects the overall energy level of the current input signal and is closely related to the signal-to-noise ratio of the input signal.

[0036] In the conditional signal mapping sub-step, the mean E is mapped to a conditional signal using the Sigmoid function. t : in, s The sigmoid function is used; the clamp function is used to truncate the calculated value within [...]. x 1,1- x 1]; x 0 is the numerical stability constant for logarithmic operations; x 1 is the cutoff constant. The specific mapping method is as follows: after adding a numerical stability constant to the mean E, take the natural logarithm, then compress it to the interval between 0 and 1 through the Sigmoid function, and finally limit the obtained value to the preset upper and lower limits through the cutoff function.

[0037] Through this mapping process, input signals at different energy levels are mapped into different conditional signals. t value t The condition signal is ∈(0,1), allowing the network to perceive the signal-to-noise ratio characteristics of the input signal. In the modulation vector generation sub-step, the condition signal is... t Logarithmic transformation, Gaussian-Fourier projection, and fully connected network mapping are performed sequentially to generate the modulation vector. c Gaussian Fourier projection uses a fixed random weight vector that is not updated after initialization to project the scalar conditional signal into a high-dimensional space. The resulting feature vector is processed by a two-layer fully connected network, with the intermediate layer using the Swish activation function, and the final output is a modulation vector. c It is used to conditionally modulate each residual block in the drift field network.

[0038] Wherein, the logarithmic transformation t log =log( t The Gaussian Fourier projection e fourier : Where W is a fixed random weight vector (not updated after initialization); the modulation vector is the final output of the two fully connected layers. c : Where Linear is a fully connected layer; swish is an activation function; e fourier It is a Gaussian Fourier projection.

[0039] Step S3: Input the compressed complex spectrum into the pre-trained drift field network, and obtain the enhanced spectrum estimate through a single forward propagation of the drift field network, using the modulation vector as the conditional input.

[0040] First, compress the complex spectrum. Y ( f , u The part is separated into real and imaginary parts, expanded into a multi-channel real-valued tensor Z, which serves as the input to the drift field network, i.e., Z = [Re( Y ),Im( Y )], where Re and Im are the real and imaginary parts, respectively, and Z is F × T The size is a real-valued tensor. The encoder part of the drift-field network performs downsampling operations step-by-step to extract feature representations at multiple resolution levels. In intermediate resolution levels, a self-attention mechanism is introduced to capture long-range dependencies between different locations in the time-frequency domain. The decoder part performs upsampling operations step-by-step to gradually restore spatial resolution. Feature concatenation between encoder levels and corresponding decoder levels is achieved through skip connections, realizing the fusion of multi-scale features. Within each residual block, the modulation vector... c After processing with the Swish activation function, the signal is mapped to the output channel number of the current residual block via a linear projection layer, and then injected into the feature map of the residual block with an additive bias. Due to the modulation vector... c From conditional signals t Generation, and conditional signals t Determined by the energy level of the input signal, input signals with different signal-to-noise ratios will acquire different additive biases in each residual block, enabling the network to perform differentiated feature modulation under different noise conditions. The final output feature map of the drift-field network, after adaptive scaling, is converted to a channel count of 2 through a convolutional layer. The two channels correspond to the real and imaginary parts of a complex number, respectively, and are assembled into a complex form to obtain the residual correction. O The residual correction is added element-wise to the compressed complex spectrum of the input to obtain the enhanced spectrum estimate. : In the initial training phase, the weights of the network's output layer are initialized to zero, and the residual correction is close to zero. The network learns and optimizes the residual correction gradually from the noisy input. Both the training and inference phases perform the exact same single forward propagation process, eliminating the problem of inconsistent training and inference distributions in multi-step integration methods.

[0041] Step S4: Convert the enhanced spectrum estimate into a time-domain signal and output the enhanced speech.

[0042] The enhanced spectral estimate of the above output is then subjected to inverse nonlinear power-law compression. Specifically, the enhanced spectral estimate is divided by a scaling factor. α Then, take 1 / of the amplitude. p The magnitude is increased by a power to restore the amplitude of the original dynamic range while keeping the phase information unchanged. Because... α For a positive real scaling factor, divide by α By changing only the amplitude without altering the phase, the phase after inverse compression becomes the original phase for the enhanced spectrum estimation. Simultaneously, the phase information remains unchanged. Then, an inverse short-time Fourier transform is performed on the recovered complex spectrum, using the exact same window function, FFT points, frame shift, and other parameters as in step S1, to obtain a normalized enhanced waveform. Finally, the peak amplitude factor saved in step S1 is used to perform amplitude recovery on this normalized waveform, yielding the final output enhanced speech.

[0043] The training mechanism that enables the aforementioned single-step reasoning ability is explained in detail below.

[0044] In the multi-scale feature extraction stage, multi-scale feature vectors are extracted from the enhanced spectrum estimate output by the above steps and the clean compressed spectrum obtained by the same processing as step S1 from the clean speech signal paired with the noisy speech signal in the training set, respectively, for use in subsequent drift field calculation.

[0045] Multi-scale feature vectors can be extracted using at least one of the following methods: manual statistical feature extraction; extraction of statistics of each hidden state as multi-scale features using a pre-trained self-supervised speech model, including but not limited to WavLM, wav2vec 2.0, and HuBERT; or a combination of manual statistical features and features extracted by a pre-trained self-supervised speech model.

[0046] This embodiment employs a manual statistical feature extraction scheme, extracting statistical feature vectors at multiple scales in both the original spectral domain and the Mel spectral domain, forming a complementary feature space across two domains.

[0047] In the original spectral domain, the complex spectrum is expanded into two real-valued tensors with real and imaginary parts. From these tensors, feature vectors of seven scales are extracted, including global statistics, frequency-dimensional statistics (including mean and standard deviation), time-dimensional statistics (including mean and standard deviation), time-frequency block mean, and channel energy. The original spectral domain features preserve phase information and a fine frequency structure, ensuring signal fidelity.

[0048] In the Mel spectral domain, the amplitude spectrum is calculated from the complex spectrum, transformed into a logarithmic Mel spectrum using a triangular Mel filter bank, and feature vectors of seven scales are extracted from it, including band statistics (mean and standard deviation), frame statistics (mean and standard deviation), Mel block statistics, and temporal difference statistics (mean and standard deviation). Mel spectral domain features are aligned with the critical frequency band perception characteristics of the human ear, optimizing auditory perception quality.

[0049] Total extraction of dual domains S =14 eigenvectors at different scales, obtaining eigenvectors at each scale and subscript S =1,…, S ; D S For the first S The feature dimensions are measured at multiple scales. The dual-domain complementary mechanism ensures signal fidelity through the original spectral domain features, while the Mel spectral domain features optimize perceptual quality. Both play a joint role in drift field calculation.

[0050] The training of the drift-field network employs a drift-field-driven parameter update mechanism. In one training iteration, a batch of noisy speech signals is first processed in step S1 to obtain a compressed complex spectrum, and the corresponding clean speech signals are processed in the same way to obtain a clean compressed spectrum. After the compressed complex spectrum generates a modulation vector in step S2, it is input into the drift-field network, and after a single forward propagation, an enhanced spectrum estimate is output. Subsequently, the drift field is constructed in the feature space. First, multi-scale feature vectors are extracted from both the enhanced spectrum estimate and the clean compressed spectrum. In this embodiment, the real and imaginary parts of the complex spectrum are expanded into two-channel real-valued tensors in the original spectral domain. Features at seven scales are extracted from these tensors: global statistics, frequency mean and standard deviation, time mean and standard deviation, time-frequency block mean, and channel energy, preserving the signal's phase information and fine frequency structure. In the Mel spectral domain, the amplitude spectrum is calculated from the complex spectrum and transformed into a logarithmic Mel spectrum using a triangular Mel filter bank. Features at seven scales—band mean and standard deviation, frame mean and standard deviation, Mel block mean, and time difference mean and standard deviation—are extracted from this spectrum to align with the human ear's critical frequency band perception characteristics. A total of fourteen feature vectors at fourteen scales are extracted across both domains, forming a complementary feature space.

[0051] In the extracted multi-scale feature space, a drift field is constructed for each feature vector x of the enhanced spectral estimation. This drift field V (p,q) (x) is attracted by the term and exclusion term It consists of two parts, and its mathematical expression is: , where x is from the generating distribution q The feature vector of the enhanced sample. Attraction term. Calculate x pointing from the clean speech distributionp Multiple positive sample features y + The weighting direction of x is used to pull x toward a clean speech distribution, and its calculation method is as follows: , where y j + For the j-th positive sample feature, k Z is a softmax affinity kernel function based on scaled L2 distance. p (x) is the corresponding normalization factor. Exclusion term. Calculate x pointing from the generating distribution q Other enhanced sample features y - The weighted direction pushes x away from the generative distribution to prevent modal collapse, and its calculation method is as follows: , where y k - Indicates the first k Each negative sample feature, Z q (x) represents the corresponding normalization factor. The drift field calculation achieves bidirectional normalization coupling through bidirectional softmax, with the row direction ensuring that each enhanced sample is fully guided and the column direction ensuring that each clean sample plays a supervisory role.

[0052] The mechanism for collaborative work between drift field training and single-step inference in this embodiment is as follows. In each training iteration, the attraction term calculates a direction vector that pulls the generated samples towards the clean distribution based on the difference between the current generated distribution and the clean distribution; the repulsion term calculates a direction vector that pushes the samples away from each other to prevent mode collapse based on the relative positions of the generated samples. After vector synthesis of the two direction vectors, a drift vector field is formed that drives the evolution of the generated distribution towards the clean distribution. Driven by this drift vector field, the drift field network updates its parameters, so that the forward propagation function of the network gradually internalizes this distribution evolution process. When training converges, the generated distribution coincides with the clean distribution under the action of the drift field, the drift field approaches zero, and the system reaches an equilibrium state. At this time, the forward propagation of the network is equivalent to completing a single-step mapping from the noisy distribution to the clean distribution. This training-inference closed-loop mechanism makes a single forward propagation in the inference stage equivalent in effect to the multi-step iterative process of traditional methods, eliminating the inherent problem of inconsistent training and inference distributions from the principle level. The energy adaptive condition and drift field network further enhance the above effects: conditional signal t Determined by the overall energy level of the input signal, strong noise and weak noise signals correspond to different... t Value, the generated modulation vector c Therefore, the network layers perform differentiated feature modulation on inputs with different signal-to-noise ratios, enabling a single forward propagation to cover the enhancement needs of the entire scenario.

[0053] A hybrid loss scheme is used during the training phase, with a total training loss of [missing information].L Loss due to drift Reconstruction loss and multi-resolution STFT loss The total training loss is calculated by weighted summation of the three factors. ,in, oh d , oh r , oh s These are the weighting coefficients for each loss term. The drift loss is constructed based on the drift field vector through a stopping gradient mechanism, applying regularization at the distribution level. The reconstruction loss calculates the logarithmic magnitude mean square error and complex mean square error for the enhanced spectrum estimate and the clean compressed spectrum, respectively, and then performs a weighted combination. The multi-resolution short-time Fourier transform loss converts the enhanced spectrum estimate into a waveform through a differentiable inverse transform, and then calculates the spectral convergence loss and logarithmic magnitude L1 loss under multiple sets of different FFT parameters. The three losses are functionally complementary, enabling the network to achieve enhancement metrics comparable to multi-step methods even under single-step output constraints. Training uses the Adam optimizer, combined with gradient accumulation to expand the equivalent batch size. The learning rate scheduling adopts a cosine annealing strategy with preheating, and the network parameters are maintained by an exponential moving average. The parameters after the exponential moving average are used during the inference phase.

[0054] The enhanced spectrum estimate of the above output is then subjected to the inverse operation of the short-time Fourier transform to recover the complex spectrum in the short-time Fourier transform domain. : ,in, α It is a positive real scaling factor, divided by α Only the amplitude is changed, not the phase. That is, the inverse nonlinear power-law compression operation involves dividing the enhanced spectral estimate by the scaling factor. α Then, take 1 / of the amplitude. p The magnitude is increased by a power to restore the amplitude of the original dynamic range while keeping the phase information unchanged. Because... α For a positive real scaling factor, divide by α By changing only the amplitude without changing the phase, the phase after inverse compression is the original phase for enhanced spectrum estimation.

[0055] The complex spectrum in the short-time Fourier transform domain obtained by inverse compression recovery is then subjected to an inverse short-time Fourier transform. This inverse transform uses the exact same window function, FFT points, frame shift, and padding parameters as the aforementioned short-time Fourier transform to obtain a normalized enhanced waveform. Finally, the peak amplitude factor saved in step S1.1 is used. c Amplitude recovery is performed on the normalized waveform to obtain the final enhanced speech output. : .

[0056] In a preferred embodiment, the drift field network adopts the NCSN++ architecture, featuring a symmetrical encoder-decoder structure. A self-attention mechanism is introduced at intermediate resolution layers, and feature fusion is achieved between layers via skip connections. Each residual block employs a BigGAN residual block structure, and the modulation vector... c After processing by the Swish activation function, the number of output channels is mapped to the current residual block via a linear projection layer. C out The feature map of the residual block is then injected with an additive bias. h : ; Due to conditional signals t Determined by the input energy, input signals with different signal-to-noise ratios receive different additive biases in each residual block. c' This allows the drift-field network to perform differentiated processing for different noise conditions. The final feature map of the drift-field network. h final First divide by the condition signal t Adaptive scaling is performed, followed by a 1×1 convolution to convert the number of channels from 4 to 2 (corresponding to the real and imaginary parts of the complex number), and then assembled into a complex output to obtain the residual correction. O .

[0057] To verify the technical effectiveness of this invention, comparative experiments were conducted on the VoiceBank dataset, a publicly available dataset in the field of speech enhancement, as shown in Table 1. Experimental results show that the method of this invention achieves a PESQ of 3.071 and an ESOI of 0.875 under the condition of requiring only single-step inference, which is essentially on par with the results of FlowSE's five-step inference. In terms of inference speed, the RTF of the method of this invention is only 0.016, while FlowSE's RTF is 0.078, making the inference speed approximately 4.8 times that of FlowSE. These comparative results demonstrate that this invention internalizes the distribution evolution process into the training phase through a drift field training mechanism, achieving enhancement quality on par with the most advanced multi-step inference methods while reducing the computational load by several times. This effect is unexpected by those skilled in the art.

[0058] Table 1: Comparison of the effects of the present invention and traditional methods.

[0059]

[0060] In another embodiment of the present invention, a speech enhancement system employing the above method is provided, comprising: a time-frequency conversion module for converting a noisy speech signal into a compressed complex spectrum; an energy adaptive condition module for generating a modulation vector based on the average amplitude of the compressed complex spectrum; a drift field network processing module for receiving the compressed complex spectrum and the modulation vector, and obtaining an enhanced spectrum estimate through a single forward propagation; and a signal reconstruction module for converting the enhanced spectrum estimate into enhanced speech. During the training phase, the system may further include a multi-scale feature extraction module, a positive sample feature library module, a drift field calculation module, and a hybrid loss module.

[0061] Although the specification has provided a detailed description, it should be understood that various changes, substitutions, and modifications can be made without departing from the spirit and scope of the invention as defined by the appended claims. Furthermore, the specific embodiments described are not intended to limit the scope of the invention, and those skilled in the art will readily understand based on this invention that existing or future-developed processes, machines, manufactures, compositions of matter, means, methods, or steps can perform substantially the same functions or achieve substantially the same results as the embodiments of the invention. Therefore, the appended claims are intended to include such processes, machines, manufactures, compositions of matter, means, methods, or steps within their scope.

Claims

1. A single-step speech enhancement method based on a drift model, characterized in that: include: Obtain the compressed complex spectrum of a noisy speech signal; A conditional signal is generated based on the average amplitude of the compressed complex spectrum, and a modulation vector is generated based on the conditional signal. The compressed complex spectrum is input into a pre-trained drift field network. Through a single forward propagation of the drift field network, the modulation vector is used as a conditional input to obtain an enhanced spectrum estimate. The enhanced spectrum estimate is converted into a time-domain signal, and the enhanced speech is output. The training of the drift field network includes: constructing a drift field V in a multi-scale feature space for the feature vector of the currently output enhanced spectral estimation. (p,q) (x) is attracted by the term and exclusion term Composition, that is , where x is from the generating distribution q The feature vector of the enhanced sample; The attraction term calculation x points to a clean speech distribution. p Multiple positive sample features y + The weighting direction pulls x toward the clean speech distribution. p The attraction item for: Among them, y j + For the first j A positive sample feature, k Z is the kernel function. p (x) is the normalization factor; The exclusion term calculation x points to the generated distribution. q Other enhanced sample features y - The weighting direction pushes x away from the generated distribution. q The exclusion term for: Among them, y k - For the first k Each negative sample feature, Z q (x) is the normalization factor; The drift field is used to drive the parameter updates of the drift field network.

2. The method according to claim 1, characterized in that, The condition signal t Generate in the following way: Calculate the mean value E of the amplitude at each time-frequency position of the compressed complex spectrum: The conditional signal is obtained by mapping the mean E as follows. t : in, F Number of frequency channels; T Y(time frame number) f , u ) represents the compressed complex spectrum at frequency f and time u The value at; σ `clamp` is the sigmoid function; `clamp` is the truncation function. ξ 0 is the numerical stability constant for logarithmic operations; ξ 1 is the cutoff constant.

3. The method according to claim 2, characterized in that, The modulation vector is generated based on the condition signal. γ ,include: For the condition signal t Taking the logarithm, we get t log =log( t ); right t log Perform Gaussian Fourier projection to obtain the projection feature e fourier : Where W is a fixed random weight vector; The projection feature e fourier Mapped to the modulation vector via a fully connected network γ : In this context, Linear is a fully connected layer, and swish is an activation function.

4. The method according to claim 1, characterized in that, The drift field network adopts the NCSN++ architecture, which features an encoder-decoder structure, skip connections, and a self-attention mechanism.

5. The method according to claim 1, characterized in that, The process of obtaining the enhanced spectrum estimate includes: The residual correction is output through a single forward propagation of the drift field network. O ; The residual correction amount O With the compressed complex spectrum Y Add them together to obtain the enhanced spectrum estimate. : 。 6. The method according to claim 1, characterized in that, The training of the drift field network also includes: Integrating drift loss using a weighted summation method Reconstruction loss and multi-resolution STFT loss To obtain the total training loss L : in, ω d , ω r , ω s These are the weighting coefficients for each type of loss; Based on the total training loss L Update the parameters of the drift field network.

7. The method according to claim 6, characterized in that, The feature vectors of the multi-scale feature space are extracted using at least one of the following methods: Manual statistical features are used to extract statistics at multiple scales in the original spectral domain and / or the Mel spectral domain. Features extracted from a pre-trained self-supervised speech model.

8. A single-step speech enhancement system employing the method described in claim 1, characterized in that, include: The time-frequency conversion module is used to convert noisy speech signals into compressed complex spectra; An energy adaptive condition module is used to generate a modulation vector based on the average amplitude of the compressed complex spectrum; A drift field network processing module is used to receive the compressed complex spectrum and the modulation vector, and obtain an enhanced spectrum estimate through a single forward propagation. A signal reconstruction module is used to convert the enhanced spectrum estimate into enhanced speech.

9. The system according to claim 8, characterized in that, The drift field network processing module adopts the NCSN++ architecture, which features an encoder-decoder structure, skip connections, and a self-attention mechanism.

10. The system according to claim 8, characterized in that, It also includes the following modules used during the training phase: The multi-scale feature extraction module is used to extract multi-scale feature vectors from the enhanced spectrum estimation and the clean compressed spectrum, respectively. The positive sample feature library module is used to accumulate multi-scale feature vectors of clean speech across batches. The drift field calculation module is used to calculate the drift field vector consisting of attraction and repulsion terms in the normalized feature space, based on the multi-scale feature vector and the features in the positive sample feature library. The hybrid loss module is used to integrate drift loss in a weighted summation manner. Reconstruction loss and multi-resolution STFT loss To obtain the total training loss L .

Images