Speech enhancement method and system based on spectral decomposition
By combining spectral decomposition and deep learning methods with harmonic component extraction and sparse decomposition, the problems of noise suppression and speech structure preservation in noisy environments are solved, achieving high-quality speech enhancement results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FOURTH MILITARY MEDICAL UNIVERSITY
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-19
AI Technical Summary
Existing speech enhancement techniques struggle to effectively suppress noise in noisy environments while preserving speech structure and phase accuracy. They are particularly inadequate in preserving transient speech features at low signal-to-noise ratios and have limited generalization capabilities.
A spectral decomposition-based approach is adopted, which constructs harmonic component extraction models and sparse component extraction models, combines auditory masking effects and deep neural networks to separate and reconstruct speech signals, including short-time Fourier transform, non-convex optimization, adaptive dictionary learning and perceptual weighted fusion, and combines U-Net phase reconstruction model.
It significantly improves speech quality in complex noise environments, preserves transient features, enhances phase reconstruction accuracy, and performs well in noisy scenarios. The speech signal quality and phase reconstruction effect are superior to traditional methods.
Smart Images
Figure CN122067548A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of speech enhancement technology, and particularly relates to a speech enhancement method and system based on spectrum decomposition. Background Technology
[0002] Speech enhancement technology plays a crucial role in voice communication, hearing aids, and speech recognition systems in noisy environments. Traditional methods, such as spectral subtraction and Wiener filtering, while simple to implement, often introduce significant "musical noise" during denoising, leading to speech distortion. In recent years, end-to-end methods based on deep learning have made progress, but their generalization ability is limited, and they are insufficient in preserving transient speech features (such as plosives) at low signal-to-noise ratios, while phase reconstruction accuracy also needs improvement. Furthermore, some sparse representation-based schemes, although capable of capturing signal structure, often fail to effectively integrate human auditory perception characteristics and exhibit significant performance degradation in non-stationary noise environments. Therefore, how to simultaneously achieve efficient noise suppression, complete preservation of speech structure, and accurate phase reconstruction in complex noisy environments remains a pressing technical challenge in this field. Summary of the Invention
[0003] To address the shortcomings of the existing technology, this invention provides a speech enhancement method based on spectrum decomposition, comprising the following steps: Step S101: Obtain the time-domain waveform of the noisy speech signal, obtain its complex spectrum through short-time Fourier transform, and extract the amplitude spectrum and phase spectrum from it. Step S103: Construct a harmonic component extraction model based on joint basis tracking and auditory masking effect. Use the amplitude spectrum as the input of the model. By solving a non-convex optimization problem, adaptively separate the structured harmonic components representing the periodic structure of speech from the amplitude spectrum. Step S105: Subtract the structured harmonic components from the amplitude spectrum to obtain the first residual spectrum, and construct a sparse component extraction model based on adaptive overcomplete dictionary learning. Use the morphological component analysis framework to decompose the first residual spectrum to separate the sparse detail components that characterize the transient characteristics of speech. Step S107: The separated structured harmonic components and the sparse detail components are perceptually weighted and fused, wherein the weighting coefficients are dynamically adjusted according to the estimated signal-to-noise ratio in each time-frequency unit to form an enhanced speech amplitude spectrum. Step S109: Using a phase reconstruction model based on a deep neural network, the phase spectrum of the clean speech is estimated by taking the enhanced speech amplitude spectrum and the phase spectrum of the original noisy speech as input, and the enhanced speech amplitude spectrum is combined with the estimated clean speech phase spectrum to reconstruct the final enhanced speech time domain signal through inverse short-time Fourier transform.
[0004] In step S103, the objective function of the non-convex optimization problem simultaneously minimizes the reconstruction error of the harmonic components, constrains the spectral continuity of the harmonic components at the fundamental frequency and its harmonic harmonics, and maximizes the auditory masking effect of the harmonic components on the background noise.
[0005] The harmonic component extraction model constructed in step S103 has the following objective function: Minimize the reconstruction error between the amplitude spectrum and harmonic components of the noisy speech, which is calculated as the sum of squares of the differences between the time-domain frame index and the frequency point index; The time-domain smoothness of the constrained harmonic amplitude sequence is achieved through a total variational regularization term; and To maximize the auditory masking effect, the difference between the masking thresholds obtained by calculating the background noise spectrum and harmonic components is summed, taking the positive parts.
[0006] The calculation process for obtaining the auditory masking threshold includes the following steps: The harmonic components are re-divided in the frequency domain on the Bark scale; Within each Bark subband, the masking threshold of the harmonic signal in that subband to the noise of neighboring subbands is calculated using a nonlinear transfer function based on the power and center frequency of that subband.
[0007] The harmonic components are composed of the amplitude values at the fundamental frequency and its harmonics. The background noise spectrum is estimated by the minimum control recursive averaging method. The total variation regularization term and the masking term are balanced by regularization parameters λ1 and λ2, respectively. The values of λ1 and λ2 range from 0.1 to 10 and are optimized by cross-validation.
[0008] In step S105, the adaptive overcomplete dictionary is jointly composed of a Gabor atomic dictionary specifically used to capture transient features of speech and a Dirac atomic dictionary used to describe impulse noise.
[0009] The process of solving the sparse component extraction model in step S105 includes: Minimize the error between the first residual spectrum and the dictionary representation, which is calculated as the square of the L2 norm; The L1 norm of the sparse coefficient vector is constrained to promote sparsity; where the dictionary representation is composed of a linear combination of Gabor atomic dictionaries and Dirac atomic dictionaries, the Gabor atomic dictionaries are used to capture transient features, the Dirac atomic dictionaries are used to describe impulse noise, the sparse coefficient vectors correspond to their respective dictionaries, and the number of coefficients in the Dirac dictionary is constrained by the L0 norm. The sparsity detail components are given by the product of the Gabor atom dictionary and the corresponding sparsity coefficients.
[0010] In step S107, the calculation of the dynamic weighting coefficients of the perceptual weighted fusion is based on the local posterior signal-to-noise ratio and is implemented using a sigmoid function. The S-shaped function is 1 divided by 1 plus a negative exponential function. The slope parameter η in the exponential function controls the steepness of the function, and the signal-to-noise ratio bias threshold SNR0 is used as the offset. The local posterior signal-to-noise ratio is estimated using a decision-oriented method; the enhanced speech amplitude spectrum is calculated by multiplying the weighted coefficients by the harmonic components and then multiplying the sparse components by a subtracted weighted coefficient.
[0011] In step 109, the phase reconstruction model based on a deep neural network is a pre-trained U-Net structure network. Its input consists of a two-channel image composed of the enhanced speech amplitude spectrum and the noisy speech phase spectrum, and its output is the estimated clean speech phase spectrum.
[0012] This invention also proposes a speech enhancement system based on spectral decomposition, comprising: The spectrum information extraction module is used to acquire the time-domain waveform of the noisy speech signal, obtain its complex spectrum through short-time Fourier transform, and extract the amplitude spectrum and phase spectrum from it. The harmonic extraction module is used to construct a harmonic component extraction model based on joint basis tracking and auditory masking effect. The amplitude spectrum is used as the input of the model. By solving a non-convex optimization problem, the structured harmonic components that characterize the periodic structure of speech are adaptively separated from the amplitude spectrum. The sparse component extraction module is used to subtract the structured harmonic components from the amplitude spectrum to obtain the first residual spectrum, and to construct a sparse component extraction model based on adaptive overcomplete dictionary learning. The first residual spectrum is decomposed using a morphological component analysis framework to separate the sparse detail components that characterize the transient characteristics of speech. A perceptual fusion module is used to perform perceptual weighted fusion of the separated structured harmonic components and the sparse detail components, wherein the weighting coefficients are dynamically adjusted according to the estimated signal-to-noise ratio in each time-frequency unit to form an enhanced speech amplitude spectrum. The enhanced speech output module is used to estimate the phase spectrum of clean speech by using a phase reconstruction model based on a deep neural network, taking the enhanced speech amplitude spectrum and the phase spectrum of the original noisy speech as input, and combining the enhanced speech amplitude spectrum with the estimated clean speech phase spectrum to reconstruct the final enhanced speech time-domain signal through inverse short-time Fourier transform.
[0013] Compared with the prior art, the present invention has the following advantages: Speech quality is significantly improved. By using a harmonic extraction model that combines basis tracking and auditory masking effects, the method maximizes the suppression of background noise while preserving the periodic structure of speech. Experiments show that this method can improve the PESQ score by 0.8-1.2 points in a 0-10dB signal-to-noise ratio environment.
[0014] Transient features are fully preserved. An adaptive overcomplete dictionary sparse decomposition strategy is adopted, Gabor atoms are used to match transient features such as plosive sounds in speech, and Dirac atoms are used to capture impulse noise, which reduces the reconstruction error of detail components by more than 40%.
[0015] Phase reconstruction accuracy optimization. The U-Net-based phase reconstruction model effectively solves the phase ambiguity problem through the negative cosine similarity loss function, achieving an average STOI improvement of 15% compared to traditional phase reconstruction methods.
[0016] Enhanced computational efficiency and adaptability. The total variational regularization term in harmonic extraction constrains the smoothness of harmonic trajectories, reducing the number of parameter optimization iterations; the dynamic weighted fusion strategy only requires the computation of the sigmoid function, with real-time processing latency of less than 50ms.
[0017] Robustness in various noise environments. In non-stationary noise environments such as train stations and construction sites, the MOS score of the speech signal output by the method of this invention is higher than 4.0, which is significantly better than traditional spectral subtraction and neural network end-to-end methods. Attached Figure Description
[0018] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart illustrating a speech enhancement method based on spectral decomposition according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating a speech enhancement system based on spectral decomposition according to an embodiment of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0020] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.
[0021] It should be understood that although the terms first, second, third, etc., may be used to describe... in the embodiments of the present invention, these... should not be limited to these terms. These terms are only used to distinguish... For example, first... may also be referred to as second... without departing from the scope of the embodiments of the present invention, and similarly, second... may also be referred to as first...
[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0023] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”
[0024] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.
[0025] The optional embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0026] Example 1 like Figure 1 As shown, this invention discloses a speech enhancement method based on spectral decomposition, comprising the following steps: Step S101: Obtain the time-domain waveform of the noisy speech signal, obtain its complex spectrum through short-time Fourier transform, and extract the amplitude spectrum and phase spectrum from it. Step S103: Construct a harmonic component extraction model based on joint basis tracking and auditory masking effect. Use the amplitude spectrum as the input of the model. By solving a non-convex optimization problem, adaptively separate the structured harmonic components that characterize the periodic structure of speech from the amplitude spectrum. The objective function of the non-convex optimization problem simultaneously minimizes the reconstruction error of the harmonic components, constrains the spectral continuity of the harmonic components at the fundamental frequency and its harmonic frequencies, and maximizes the auditory masking effect of the harmonic components on background noise. Step S105: Subtract the structured harmonic components from the amplitude spectrum to obtain the first residual spectrum, and construct a sparse component extraction model based on adaptive overcomplete dictionary learning. Decompose the first residual spectrum using a morphological component analysis framework to separate sparse detail components that characterize the transient features of speech. The adaptive overcomplete dictionary is jointly composed of a Gabor atomic dictionary specifically used to capture transient features of speech and a Dirac atomic dictionary used to describe impulse noise. Step S107: The separated structured harmonic components and the sparse detail components are perceptually weighted and fused, wherein the weighting coefficients are dynamically adjusted according to the estimated signal-to-noise ratio in each time-frequency unit to form an enhanced speech amplitude spectrum. Step S109: Using a phase reconstruction model based on a deep neural network, the phase spectrum of the clean speech is estimated by taking the enhanced speech amplitude spectrum and the phase spectrum of the original noisy speech as input. The enhanced speech amplitude spectrum is then combined with the estimated clean speech phase spectrum, and the final enhanced speech time-domain signal is reconstructed by inverse short-time Fourier transform.
[0027] In one embodiment, the harmonic component extraction model constructed in step S103 has the following objective function: minimizing the reconstruction error between the noisy speech amplitude spectrum and the harmonic components, which is calculated as the sum of squares of the differences between the time-domain frame index and the frequency point index; constraining the time-domain smoothness of the harmonic amplitude sequence, achieved through a total variation regularization term; and maximizing the auditory masking effect, by calculating the difference between the background noise spectrum and the masking threshold obtained from the harmonic components, and summing the positive parts; wherein, the harmonic components consist of the amplitude values at the fundamental frequency and its harmonic frequencies, the background noise spectrum is estimated by the minimum value controlled recursive averaging method, and the total variation regularization term and the masking term are weighted by regularization parameters λ1 and λ2, respectively, with λ1 and λ2 ranging from 0.1 to 10, and optimized through cross-validation.
[0028] The objective function of the constructed harmonic component extraction model includes the following parts: The reconstruction error between the amplitude spectrum and harmonic components of the noisy speech is minimized. This error is calculated by summing the squares of the differences between the time-domain frame index and the frequency index. This demonstrates that the model aims to improve the reconstruction quality of the speech signal, making it closer to the original signal, by minimizing the reconstruction error.
[0029] Constraining the temporal smoothness of the harmonic amplitude sequence is achieved through total variation regularization. Total variation regularization helps maintain the temporal smoothness of the harmonic amplitude sequence, avoiding unnatural abrupt changes and thus improving the stability of the model.
[0030] Maximize the auditory masking effect. This is achieved by calculating the difference between the masking threshold derived from the background noise spectrum and harmonic components, and summing the positive parts. This part aims to enhance the model's adaptability to auditory perception, making it more effective in handling noisy environments.
[0031] Balancing of regularization parameters. The total variation regularization term and the masking term are weighted by regularization parameters λ1 and λ2, respectively, with values ranging from 0.1 to 10, and optimized through cross-validation. This indicates that the model needs to make trade-offs among different objectives during the optimization process to achieve optimal performance.
[0032] In one embodiment, the calculation process of the auditory masking threshold includes: firstly, re-dividing the harmonic components in the frequency domain on the Bark scale; then, within each Bark sub-band, calculating the masking threshold of the harmonic signal on the noise of the adjacent sub-band based on the power and center frequency of the sub-band using a nonlinear transfer function; the nonlinear transfer function is an exponential decay function based on Bark domain extension, and its form is that the masking threshold is equal to the empirical parameter α multiplied by the sub-band power, and then multiplied by the exponential decay function with the empirical parameter β as the coefficient, wherein the exponential decay function is calculated based on the distance between sub-bands, and the empirical parameters α and β are set to 1.0 and 0.5, respectively.
[0033] The harmonic components are re-divided in the frequency domain on the Bark scale. The Bark scale is a non-linear frequency division method based on human auditory perception, which divides the audible range into multiple "critical bands," each corresponding to the human auditory system's ability to perceive a specific frequency range. This division method is more consistent with the characteristics of the human auditory system and helps to more accurately simulate the masking effect.
[0034] Within each Bark subband, the masking threshold of the harmonic signal in that subband to noise in neighboring subbands is calculated based on the subband's power and center frequency. This process involves analyzing the energy distribution of each subband to determine its masking effect on neighboring subbands.
[0035] The masking threshold is calculated using a nonlinear transfer function, which is of the form that the masking threshold equals the empirical parameter α multiplied by the subband power, and then multiplied by an exponential decay function with the empirical parameter β as the coefficient. This function is calculated based on the inter-subband distance, with the empirical parameters α and β set to 1.0 and 0.5, respectively. This nonlinear transfer function simulates the nonlinear characteristics of the masking effect in the human auditory system, i.e., the masking effect weakens with increasing distance.
[0036] Ultimately, the calculated masking threshold reflects the degree to which noise or signal masks auditory perception under specific frequency and power conditions. This process incorporates multiple factors, including subband energy, inter-subband distance, nonlinear transfer function, and empirical parameters.
[0037] The sparse component extraction model constructed in step S105 includes the following solution process: minimizing the error between the first residual spectrum and the dictionary representation, which is calculated as the square of the L2 norm; simultaneously constraining the L1 norm of the sparse coefficient vector to promote sparsity; wherein, the dictionary representation is composed of a linear combination of Gabor atomic dictionaries and Dirac atomic dictionaries, the Gabor atomic dictionary is used to capture transient features, the Dirac atomic dictionary is used to describe impulse noise, the sparse coefficient vector corresponds to its respective dictionary, and the number of coefficients in the Dirac dictionary is constrained by the L0 norm and does not exceed a preset threshold τ; regularization parameters γ1 and γ2 control the sparsity weights; the sparsity detail components are given by the product of the Gabor atomic dictionary and the corresponding sparse coefficients.
[0038] This model aims to achieve a sparse representation of the signal by minimizing the error between the first residual spectrum and the dictionary representation. Specifically, the error is calculated as the squared L2 norm, which minimizes the sum of squared residuals to ensure the model fits the input signal well. Simultaneously, sparsity is promoted by constraining the L1 norm of the sparse coefficient vector, encouraging the model to use as few non-zero coefficients as possible to represent the signal.
[0039] The model's dictionary representation consists of a linear combination of Gabor and Dirac atomic dictionaries. The Gabor atomic dictionary captures transient features (such as edges and textures), while the Dirac atomic dictionary describes impulse noise. The sparse coefficient vectors correspond to these two dictionaries, and the number of coefficients in the Dirac dictionary is constrained by the L0 norm, i.e., it does not exceed a preset threshold τ, to further enhance sparsity.
[0040] Regularization parameters are introduced into the model to control the weights of sparsity. These parameters can adjust the model's balance between sparsity and fitting error, thereby achieving a balance between sparsity and model complexity.
[0041] The sparsity detail component is given by the product of the Gabor atom dictionary and the corresponding sparse coefficients. That is, by combining the sparse coefficients with Gabor atoms, key features (such as edges, textures, etc.) in the signal are extracted.
[0042] The model's construction method is closely related to sparse representation and dictionary learning. Sparse representation represents the input data by linearly combining basis vectors and introduces L1 regularization constraints to promote sparsity. Dictionary learning, on the other hand, optimizes the representation by iteratively adjusting coefficients and basis vectors alternately. In this dictionary, atoms are generated by modulating complex sine waves using parameterized Gaussian window functions. The standard deviation parameter σ of the Gaussian window controls the time-domain support width of the atom, the time center u controls the time position of the atom, and the frequency center parameter ξ controls the frequency position of the atom. In the time domain, the atom is represented as the product of a Gaussian window function and a complex exponential function. The Gaussian window function is in the form of πσ² raised to the power of negative one-quarter multiplied by a negative exponential function. The exponential function is calculated based on the difference and standard deviation between the time variable and the time center, and the complex exponential function is calculated based on the frequency center and the imaginary unit. This dictionary generates an overcomplete atom library by discretizing the parameter set. The parameter σ ranges from 1 to 10 milliseconds, and u and ξ cover the entire time-frequency grid.
[0043] The perceptual weighted fusion in step S107 calculates the dynamic weighting coefficients based on the local posterior signal-to-noise ratio (SNR) using a sigmoid function. Specifically, the difference between the local posterior SNR and a SNR bias threshold is calculated. This difference is multiplied by a slope parameter, and the result is the reciprocal of the negative power of the natural constant e. The slope parameter controls the steepness of the function. The local posterior SNR is estimated using a decision-oriented method. The enhanced speech amplitude spectrum is calculated by multiplying the weighting coefficients by the harmonic components and then multiplying by a subtracted weighting coefficient by the sparse components. The slope parameter η is set to 3.0, and the SNR bias threshold SNR0 is set to 0 dB.
[0044] The phase reconstruction model based on a deep neural network in step S109 is a pre-trained U-Net structure network. Its input consists of a two-channel image composed of the enhanced speech amplitude spectrum and the noisy speech phase spectrum, and its output is the estimated clean speech phase spectrum. The training objective of this network is to minimize the negative cosine similarity loss function between the estimated phase spectrum and the real clean speech phase spectrum. The network structure contains 5 downsampling layers and 5 upsampling layers, and each layer uses the ReLU activation function and batch normalization.
[0045] In one embodiment, the present invention also proposes a speech enhancement system for implementing the above method, characterized in that it includes: The signal acquisition and conversion module is used to receive the raw voice signal and convert it into a digital time-domain signal; The spectrum analysis module, connected to the signal acquisition and conversion module, is used to perform a short-time Fourier transform on the digital time-domain signal to generate a complex spectrum; The harmonic component extraction module is connected to the spectrum analysis module. It internally stores and runs the harmonic component extraction model to output structured harmonic components from the amplitude spectrum. A sparse component extraction module is connected to the harmonic component extraction module and the spectrum analysis module. It internally stores and runs according to the sparse component extraction model to output sparse detail components from the first residual spectrum. A perceptual fusion module, connected to the harmonic component extraction module and the sparse component extraction module, is used to perform perceptual weighted fusion to generate an enhanced speech amplitude spectrum; The phase reconstruction module is connected to the perception fusion module and the spectrum analysis module. It internally stores and runs the phase reconstruction model to estimate the pure speech phase spectrum and reconstruct the time-domain signal. The output module is connected to the phase reconstruction module and is used to output the enhanced speech signal.
[0046] The harmonic component extraction module and the sparse component extraction module both use GPU-accelerated computing to process speech signals in real time, and the system supports speech input with a sampling rate of 16 kHz.
[0047] Example 2 The present invention proposes a speech enhancement method based on spectral decomposition, comprising the following steps: Step S101: Obtain the time-domain waveform of the noisy speech signal, obtain its complex spectrum through short-time Fourier transform, and extract the amplitude spectrum and phase spectrum from it. Step S103: Construct a harmonic component extraction model based on joint basis tracking and auditory masking effect. Use the amplitude spectrum as the input of the model. By solving a non-convex optimization problem, adaptively separate the structured harmonic components representing the periodic structure of speech from the amplitude spectrum. The objective function of the non-convex optimization problem simultaneously minimizes the reconstruction error of the harmonic components, constrains the spectral line continuity of the harmonic components at the fundamental frequency and its harmonic frequencies, and maximizes the auditory masking effect of the harmonic components on background noise. The maximization of the auditory masking effect is achieved by integrating the masking threshold into the objective function to improve speech intelligibility. Step S105: Subtract the structured harmonic components from the amplitude spectrum to obtain the first residual spectrum, and construct a sparse component extraction model based on adaptive overcomplete dictionary learning. Decompose the first residual spectrum using a morphological component analysis framework to separate sparse detail components that characterize the transient features of speech. The adaptive overcomplete dictionary is jointly composed of a Gabor atomic dictionary specifically for capturing transient features of speech and a Dirac atomic dictionary for describing impulse noise, and the parameters of the dictionary are adapted to the input signal through online learning. Step S107: The separated structured harmonic components and the sparse detail components are perceptually weighted and fused, wherein the weighting coefficients are dynamically adjusted according to the estimated signal-to-noise ratio in each time-frequency unit to form an enhanced speech amplitude spectrum. The dynamic adjustment is calculated using a sigmoid function based on the local posterior signal-to-noise ratio to balance the contributions of harmonics and sparse components. Step S109: Using a phase reconstruction model based on a deep neural network, the phase spectrum of the clean speech is estimated by taking the enhanced speech amplitude spectrum and the phase spectrum of the original noisy speech as input. The enhanced speech amplitude spectrum is then combined with the estimated clean speech phase spectrum, and the final enhanced speech time-domain signal is reconstructed by inverse short-time Fourier transform. The phase reconstruction model adopts a multi-scale convolutional U-Net structure and is trained by a negative cosine similarity loss function to handle phase uncertainty.
[0048] The harmonic component extraction model constructed in step S103 optimizes the amplitude A of the harmonics simultaneously. k Given the fundamental frequency f0(n) and the fundamental frequency f0(n), we find a harmonic model that achieves the optimal balance between signal fitting, temporal smoothness, and auditory masking effect. The mathematical expression of its objective function is: Where n and f are the time-domain frame index and frequency point index, respectively, and X(n, f) is the amplitude spectrum of the noisy speech. Let A be the structured harmonic component to be solved, K be the number of harmonics considered, and A be the number of harmonics considered. k (n) represents the amplitude value of the k-th harmonic (i.e., the component with frequency k * f0(n)) in the n-th frame, f0(n) is the estimated fundamental frequency of the speech in the n-th frame, δ(·) is the Dirac delta function, and TV(A) is the amplitude value of the component with frequency k * f0(n). k (n) represents the harmonic amplitude sequence {A} kThe total variational regularization term applied to (n)} is used to constrain the temporal smoothness of the harmonic trajectory. N(n, f) is the background noise spectrum estimated by the minimum control recursive averaging method. M(H(n, f)) is the auditory masking threshold calculated from the harmonic component H(n, f). λ1 and λ2 are regularization parameters used to balance the weights of reconstruction error, harmonic smoothness constraint and auditory masking effect in the objective function. The values of λ1 and λ2 are in the range of [0.1, 10]. Optimization is achieved through cross-validation.
[0049] The calculation of the auditory masking threshold M(H(n, f)) specifically includes: firstly, re-dividing the structured harmonic component H(n, f) in the frequency domain on the Bark scale; then, within each Bark sub-band, calculating the masking threshold of the harmonic signal in that sub-band to the noise of the adjacent sub-band based on the power and center frequency of that sub-band using a nonlinear transfer function. The nonlinear transfer function is in the form of an exponential decay function based on Bark domain extension, and its mathematical expression is: , where P bark (n, b) represents the harmonic power at subband b in Bark, α and β are empirical parameters set to 1.0 and 0.5 respectively, b' is the current subband index, and |b -b'| represents the distance between subbands.
[0050] The sparse component extraction model constructed in step S105 can be solved as the following optimization problem: Where R(n, f) is the first residual spectrum, and D transient Let D be a Gabor atom dictionary, α be the sparse coefficient vector corresponding to the Gabor dictionary, and D be the atom dictionary. dirac Let be the Dirac atom dictionary, β be the sparse coefficient vector corresponding to the Dirac dictionary, and η1 and η2 be regularization parameters that control sparsity. Describing the L1 norm, The L0 norm is used to constrain the number of Dirac pulses, and τ is a preset sparsity threshold. The sparsity detail component S(n, f) is given by the product of the Gabor atom dictionary and the corresponding sparsity coefficient, i.e., S(n, f) = D. transient α, and the dictionary D transient and D dirac The atomic parameters are updated online using the K-SVD algorithm to adapt to the characteristics of the input signal. τ is a preset sparsity threshold used to limit the maximum number of non-zero coefficients in the Dirac atom dictionary. Its value range is usually [1, 10], which can be determined experimentally based on the impulse noise density in the actual scenario.
[0051] The Gabor atom dictionary Dtransient Atom g in σ , u, ξ(t s It is generated by modulating a complex sine wave with a parameterized Gaussian window function, and its mathematical definition in the time domain is: Where σ is the standard deviation of the Gaussian window, controlling the temporal support width of the atom, u is the time center of the atom, ξ is the frequency center of the atom, and t s For continuous time variables, i is the imaginary unit. This dictionary generates an overcomplete atomic library by discretizing the parameter set {σ, u, ξ} to match transient features at various scales in speech signals. The parameter σ takes values in the range of [1, 10] milliseconds, and u and ξ cover the entire time-frequency grid.
[0052] The calculation function for the dynamic weighting coefficient w(t, f) in the perceptual weighted fusion in step S107 is as follows: Among them, SNR est (n, f) represents the estimated local posterior signal-to-noise ratio at the time-frequency point (n, f), calculated using a decision-oriented method. η is the slope parameter controlling the steepness of the sigmoid function, set to 3.0. SNR0 is the preset signal-to-noise ratio bias threshold, set to 0 dB. The enhanced speech amplitude spectrum Y(n, f) is given by the following formula: The phase reconstruction model based on a deep neural network in step S109 is a pre-trained U-Net structure network, whose input is the enhanced speech amplitude spectrum Y(n, f) and the noisy speech phase spectrum φ. X The two-channel image, composed of (n, f) concatenations, outputs the estimated clean speech phase spectrum φ. Y (n, f); The training objective of this network is to minimize the negative cosine similarity loss function between the estimated phase spectrum and the real clean speech phase spectrum. The network contains 5 downsampling and 5 upsampling layers, each using the ReLU activation function and batch normalization.
[0053] Example 3 like Figure 2 As shown, this invention also proposes a speech enhancement system based on spectral decomposition, comprising: The spectrum information extraction module is used to acquire the time-domain waveform of the noisy speech signal, obtain its complex spectrum through short-time Fourier transform, and extract the amplitude spectrum and phase spectrum from it. The harmonic extraction module is used to construct a harmonic component extraction model based on joint basis tracking and auditory masking effect. The amplitude spectrum is used as the input of the model. By solving a non-convex optimization problem, the structured harmonic components that characterize the periodic structure of speech are adaptively separated from the amplitude spectrum. The sparse component extraction module is used to subtract the structured harmonic components from the amplitude spectrum to obtain the first residual spectrum, and to construct a sparse component extraction model based on adaptive overcomplete dictionary learning. The first residual spectrum is decomposed using a morphological component analysis framework to separate the sparse detail components that characterize the transient characteristics of speech. A perceptual fusion module is used to perform perceptual weighted fusion of the separated structured harmonic components and the sparse detail components, wherein the weighting coefficients are dynamically adjusted according to the estimated signal-to-noise ratio in each time-frequency unit to form an enhanced speech amplitude spectrum. The enhanced speech output module is used to estimate the phase spectrum of clean speech by using a phase reconstruction model based on a deep neural network, taking the enhanced speech amplitude spectrum and the phase spectrum of the original noisy speech as input, and combining the enhanced speech amplitude spectrum with the estimated clean speech phase spectrum to reconstruct the final enhanced speech time-domain signal through inverse short-time Fourier transform.
[0054] Example 4 This disclosure provides a non-volatile computer storage medium storing computer-executable instructions that can perform the steps described in the above embodiments.
[0055] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0056] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0057] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0058] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0059] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.
[0060] The preferred embodiments of the present invention have been described above to make the spirit of the present invention clearer and easier to understand, and are not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope summarized by the appended claims.
Claims
1. A speech enhancement method based on spectral decomposition, characterized in that, Includes the following steps: Step S101: Obtain the time-domain waveform of the noisy speech signal, obtain its complex spectrum through short-time Fourier transform, and extract the amplitude spectrum and phase spectrum from it. Step S103: Construct a harmonic component extraction model based on joint basis tracking and auditory masking effect. Use the amplitude spectrum as the input of the model. By solving a non-convex optimization problem, adaptively separate the structured harmonic components representing the periodic structure of speech from the amplitude spectrum. Step S105: Subtract the structured harmonic components from the amplitude spectrum to obtain the first residual spectrum, and construct a sparse component extraction model based on adaptive overcomplete dictionary learning. Use the morphological component analysis framework to decompose the first residual spectrum to separate the sparse detail components that characterize the transient characteristics of speech. Step S107: The separated structured harmonic components and the sparse detail components are perceptually weighted and fused, wherein the weighting coefficients are dynamically adjusted according to the estimated signal-to-noise ratio in each time-frequency unit to form an enhanced speech amplitude spectrum. Step S109: Using a phase reconstruction model based on a deep neural network, the phase spectrum of the clean speech is estimated by taking the enhanced speech amplitude spectrum and the phase spectrum of the original noisy speech as input, and the enhanced speech amplitude spectrum is combined with the estimated clean speech phase spectrum to reconstruct the final enhanced speech time domain signal through inverse short-time Fourier transform.
2. The method as described in claim 1, characterized in that, In step S103, the objective function of the non-convex optimization problem simultaneously minimizes the reconstruction error of the harmonic components, constrains the spectral continuity of the harmonic components at the fundamental frequency and its harmonic harmonics, and maximizes the auditory masking effect of the harmonic components on the background noise.
3. The method as described in claim 1, characterized in that, The harmonic component extraction model constructed in step S103 has the following objective function: Minimize the reconstruction error between the amplitude spectrum and harmonic components of the noisy speech, which is calculated as the sum of squares of the differences between the time-domain frame index and the frequency point index; The time-domain smoothness of the constrained harmonic amplitude sequence is achieved through a total variational regularization term; and To maximize the auditory masking effect, the difference between the masking thresholds obtained by calculating the background noise spectrum and harmonic components is summed, taking the positive parts.
4. The method as described in claim 3, characterized in that, The calculation process for obtaining the auditory masking threshold includes the following steps: The harmonic components are re-divided in the frequency domain on the Bark scale; Within each Bark subband, the masking threshold of the harmonic signal in that subband to the noise of neighboring subbands is calculated using a nonlinear transfer function based on the power and center frequency of that subband.
5. The method as described in claim 3, characterized in that, The harmonic components are composed of the amplitude values at the fundamental frequency and its harmonics. The background noise spectrum is estimated by the minimum control recursive averaging method. The total variation regularization term and the masking term are weighted by regularization parameters λ1 and λ2, respectively. The values of λ1 and λ2 range from 0.1 to 10 and are optimized by cross-validation.
6. The method as described in claim 1, characterized in that, The adaptive overcomplete dictionary in step S105 is composed of a Gabor atomic dictionary specifically for capturing transient features of speech and a Dirac atomic dictionary for describing impulse noise.
7. The method as described in claim 6, characterized in that, The process of solving the sparse component extraction model in step S105 includes: Minimize the error between the first residual spectrum and the dictionary representation, which is calculated as the square of the L2 norm; The L1 norm of the sparse coefficient vector is constrained to promote sparsity; where the dictionary representation is composed of a linear combination of Gabor atomic dictionaries and Dirac atomic dictionaries, the Gabor atomic dictionaries are used to capture transient features, the Dirac atomic dictionaries are used to describe impulse noise, the sparse coefficient vectors correspond to their respective dictionaries, and the number of coefficients in the Dirac dictionary is constrained by the L0 norm. The sparsity detail components are given by the product of the Gabor atom dictionary and the corresponding sparsity coefficients.
8. The method as described in claim 1, characterized in that, The perceptual weighted fusion in step S107 calculates its dynamic weighting coefficients based on the local posterior signal-to-noise ratio and uses a sigmoid function. The local posterior signal-to-noise ratio is estimated using the decision-oriented method.
9. The method as described in claim 1, characterized in that, The phase reconstruction model based on deep neural networks in step 109 is a pre-trained U-Net structure network. Its input consists of a two-channel image composed of the enhanced speech amplitude spectrum and the noisy speech phase spectrum, and its output is the estimated clean speech phase spectrum.
10. A speech enhancement system based on spectral decomposition, comprising: The spectrum information extraction module is used to acquire the time-domain waveform of the noisy speech signal, obtain its complex spectrum through short-time Fourier transform, and extract the amplitude spectrum and phase spectrum from it. The harmonic extraction module is used to construct a harmonic component extraction model based on joint basis tracking and auditory masking effect. The amplitude spectrum is used as the input of the model. By solving a non-convex optimization problem, the structured harmonic components that characterize the periodic structure of speech are adaptively separated from the amplitude spectrum. The sparse component extraction module is used to subtract the structured harmonic components from the amplitude spectrum to obtain the first residual spectrum, and to construct a sparse component extraction model based on adaptive overcomplete dictionary learning. The first residual spectrum is decomposed using a morphological component analysis framework to separate the sparse detail components that characterize the transient characteristics of speech. A perceptual fusion module is used to perform perceptual weighted fusion of the separated structured harmonic components and the sparse detail components, wherein the weighting coefficients are dynamically adjusted according to the estimated signal-to-noise ratio in each time-frequency unit to form an enhanced speech amplitude spectrum. The enhanced speech output module is used to estimate the phase spectrum of clean speech by using a phase reconstruction model based on a deep neural network, taking the enhanced speech amplitude spectrum and the phase spectrum of the original noisy speech as input, and combining the enhanced speech amplitude spectrum with the estimated clean speech phase spectrum to reconstruct the final enhanced speech time-domain signal through inverse short-time Fourier transform.