A speech denoising and speech synthesis enhancement method and system based on deep learning

Abstract

This invention relates to the field of speech signal processing technology, specifically to a method and system for speech denoising and speech synthesis enhancement based on deep learning. It includes the following steps: complex spectrum data preprocessing; speech denoising processing, where each frame of preprocessed complex spectrum data is simultaneously input into a stationary background noise branch and a non-stationary burst noise branch for parallel denoising processing; speech synthesis enhancement processing; and complex spectrum data output. This invention guides branch allocation through a noise stationarity coefficient, diverting noisy speech frame by frame to a fixed receptive field depth separable convolutional branch and a dynamic receptive field deformable convolutional branch, achieving differentiated denoising between stationary background noise and non-stationary burst noise; and uses the same coefficient to drive a gating network to complete adaptive fusion of the dual-branch outputs, effectively overcoming the limitation of a single network structure in handling two significantly different types of noise, thus improving the accuracy and fidelity of speech denoising in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and more specifically, to a method and system for speech denoising and speech synthesis enhancement based on deep learning. Background Technology

[0002] With the rapid development of technologies such as intelligent voice interaction, remote conferencing, hearing aids, and multimedia communication, speech signal processing faces increasingly complex acoustic challenges in practical applications. Speech denoising, as a crucial step in speech signal front-end processing, aims to recover clean target speech as completely as possible from noisy speech, thereby improving intelligibility and auditory quality. In recent years, deep learning technology, with its significant advantages in feature representation and nonlinear mapping capabilities, has become the mainstream technology in the field of speech denoising. Typical deep learning speech denoising methods usually include complex spectrum data preprocessing, speech denoising processing, and speech enhancement, using deep neural networks to learn denoising mappings for noisy speech.

[0003] However, existing deep learning-based speech denoising methods still face significant performance bottlenecks in practical deployments. Noise in real acoustic environments is not a single type, but typically a mixture of two main categories: stationary background noise and non-stationary burst noise. Stationary background noise, such as fan noise and air conditioner noise, exhibits statistical characteristics that change slowly over time; non-stationary burst noise, such as keyboard clicks, door slams, and alarm sounds, has drastic and sudden changes in its time-frequency characteristics. Current technologies often employ a single network structure and a fixed receptive field design to uniformly address these two significantly different types of noise. This design struggles to achieve an optimal balance in suppressing both types of noise simultaneously. Specifically, a network structure with a fixed receptive field can effectively utilize long-term contextual information to achieve good smoothing results when dealing with stationary background noise. However, when facing non-stationary burst noise, it lacks the ability to adaptively capture local dynamic changes in noise, easily leading to insufficient denoising or residual noise. Conversely, if the network structure is optimized for burst noise, the limited receptive field or insufficient assumptions about signal stationarity may result in decreased suppression of stationary background noise and the introduction of unnatural distortion in the speech segment. In view of this, we propose a deep learning-based method and system for speech denoising and speech synthesis enhancement. Summary of the Invention

[0004] The purpose of this invention is to provide a speech denoising and speech synthesis enhancement method and system based on deep learning, in order to solve the problem that the existing deep learning speech denoising methods mentioned in the background art, due to their use of a single network structure and fixed receptive field design, are difficult to effectively suppress both stationary background noise and non-stationary burst noise under a unified processing framework, resulting in the inability to achieve the optimal balance between the two types of noise denoising effects at the same time.

[0005] To address the aforementioned technical problems, one objective of this invention is to provide a deep learning-based method for speech denoising and speech synthesis enhancement, comprising: S1. Speech data preprocessing: The original noisy speech data is subjected to frame-by-frame windowing and short-time Fourier transform to extract complex spectrum features and complete standardization to obtain preprocessed complex spectrum data. S2. Speech noise reduction processing: Extract noise stationarity features from the preprocessed complex spectrum data, and simultaneously input each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch for parallel noise reduction processing. The stationary background noise branch uses a depthwise separable convolution with a fixed receptive field to denoise the complex spectrum features, while the non-stationary burst noise branch uses a deformable convolution with a dynamic sampling offset range to denoise the complex spectrum features. By using an adaptive gating mechanism driven by noise features, adaptive weights are generated based on noise stationarity features and the outputs of the two branches are weighted and fused to obtain denoised complex spectrum data. S3. Speech Synthesis Enhancement Processing: Deep learning enhancement technology is used to perform speech synthesis enhancement processing on the denoised complex spectrum data to obtain synthesized enhanced complex spectrum data. S4. Speech Data Output: After verifying the synthesized and enhanced speech data, output the final time-domain speech data.

[0006] As a further improvement to this technical solution, the voice data preprocessing process in S1 includes the following steps: S11. Sample and normalize the original noisy speech data to unify the sampling rate and bit depth of the speech data and obtain the original noisy speech data in a standardized format. S12. Perform frame-segmentation, windowing, and short-time Fourier transform on the original noisy speech data in the standardized format to obtain the original complex spectrum feature set; S13. Perform outlier removal and dimension normalization on the original complex spectrum feature set to remove invalid feature data and obtain the normalized complex spectrum feature set. S14. Using a deep learning normalization algorithm, the amplitude and phase of the normalized complex spectrum feature set are normalized to eliminate the difference in feature dimensions and obtain preprocessed complex spectrum data.

[0007] As a further improvement to this technical solution, the speech noise reduction process in S2 includes the following steps: S21. Perform frame-by-frame noise stationarity feature extraction on the preprocessed complex spectrum data to obtain the noise stationarity feature extraction results for each frame. S22. Input each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch simultaneously, so that the two branches can perform noise reduction processing on the same frame of data in parallel. S23. The stationary background noise branch and the non-stationary burst noise branch perform noise reduction processing in parallel, and the dimensions of the complex spectrum features output by the two branches are consistent. The stationary background noise branch uses a depthwise separable convolution with a fixed receptive field to perform noise reduction processing on the complex spectrum features, and the non-stationary burst noise branch uses a deformable convolution with a dynamic sampling offset range to perform noise reduction processing on the complex spectrum features, and the non-stationary branch output complex spectrum features. S24. Based on the noise stationarity feature extraction results, an adaptive fusion weight is generated. The stationary branch output features and the non-stationary branch output features are weighted and fused through an adaptive gating mechanism driven by noise features to obtain denoised complex spectrum data.

[0008] As a further improvement to this technical solution, the process of extracting frame-by-frame noise stationarity features in step S21 includes the following steps: S21.1 Extract the preprocessed complex spectrum data frame by frame to obtain the first frame. Frame complex spectrum data; S21.2, calculate the first one respectively. Short-time energy, short-time zero-crossing rate, and spectral flatness of frame complex spectrum data; S21.3 Perform maximum-minimum normalization on the short-time energy and short-time zero-crossing rate in the [0,1] interval; S21.4. The normalized short-time energy, the normalized short-time zero-crossing rate, and the spectral flatness are weighted and summed to obtain the... Noise stationarity coefficient of frame complex spectrum data The sum of the weighting coefficients is 1; S21.5, Noise stationarity coefficients of all frames By performing temporal splicing in chronological order, the noise stationarity feature extraction results are obtained.

[0009] As a further improvement to this technical solution, in S22, based on the first The calculation of the frame noise stationarity coefficients for the dual-branch fusion weights includes the following steps: S22.1, Extracting the noise stationarity feature from the extraction results. Frame noise stationarity coefficient ; S22.2, Based on the aforementioned noise stability coefficient Determine the first The weights for the stationary and non-stationary branches corresponding to a frame are fused together, and the sum of the two fused weights is 1.

[0010] As a further improvement to this technical solution, the process of performing smooth background noise branch denoising in step S23 includes the following steps: S23.11, the first Frame preprocessing complex spectral data is input into a depth-separable convolution with a fixed receptive field; S23.12. Perform a depthwise convolution operation, performing convolution calculations on each input channel of the complex spectral features separately; S23.13 Perform pointwise convolution operation to perform channel fusion on the output of depthwise convolution, and maintain the integrity of the complex spectral feature structure; S23.14. Output the processed complex spectrum feature data as the first... The smooth branching output characteristics of the frame.

[0011] As a further improvement to this technical solution, the process of performing non-stationary burst noise branch denoising processing in step S23 includes the following steps: S23.21, according to the... Frame noise stationarity coefficient Determine the normalized intensity level of the non-stationary burst noise in the nth frame; S23.22, Based on noise stationarity coefficient The effective sampling coverage of deformable convolution is dynamically adjusted by adjusting the amplitude range of the sampling offset without changing the fixed receptive field size of the convolution kernel. S23.23, Based on the noise stability coefficient The normalized intensity level of non-stationary burst noise is used to dynamically adjust the kernel offset of deformable convolution, where the noise stationarity coefficient is... The smaller the value and the higher the normalization intensity level, the greater the convolution kernel offset. S23.24. Apply the adjusted deformable convolution to the first... The frame preprocessing complex spectrum data is convolved to extract and suppress non-stationary burst noise features; S23.25. Output the processed complex spectrum feature data as the first... Non-stationary branching output characteristics of frames.

[0012] As a further improvement to this technical solution, the adaptive gating fusion process in S24 includes the following steps: S24.1, according to the first Frame noise stationarity coefficient Calculate the first Frame smooth branch fusion weights Weights for merging non-stationary branches ,in , ; S24.2, An adaptive gating mechanism driven by noise characteristics is used to fuse weights according to the stable branch. Weights for merging non-stationary branches , for the The smooth branch output characteristics of the frame and the first The non-stationary branch output features of the frame are summed element-wise using a weighted summation. S24.3. The weighted summation results of all frames are fused to obtain the denoised complex spectrum data.

[0013] As a further improvement to this technical solution, the speech synthesis enhancement process in S3 includes the following steps: S3.1 Perform frame structure calibration on the denoised complex spectrum data to obtain the normalized denoised complex spectrum data; S3.2 Extract the acoustic features of the framed, denoised complex spectrum data, including fundamental frequency features, formant features and Mel spectrum features; S3.3. A deep learning enhancement model is used to process the extracted acoustic features and complex spectral features. The deep learning enhancement model uses the first... Frame noise stationarity coefficient As a conditional input, the enhancement intensity or feature fusion weights are dynamically adjusted. S3.4. Post-process the processed complex spectrum feature data to remove outliers and residual noise. S3.5. Perform inverse short-time Fourier transform and overlap summation on the post-processed synthesized enhanced complex spectrum data to obtain synthesized enhanced speech data.

[0014] A second objective of this invention is to provide a deep learning-based speech denoising and speech synthesis enhancement system. The computer program of this system, when executed, performs the steps of the aforementioned deep learning-based speech denoising and speech synthesis enhancement method, including: The speech data preprocessing unit performs frame-segmentation, windowing, and short-time Fourier transform on the original noisy speech data, extracts complex spectral features, and completes standardization processing to obtain preprocessed complex spectral data. A speech denoising processing unit performs denoising processing on preprocessed complex spectrum data to obtain denoised complex spectrum data. The speech denoising processing unit includes a noise stationarity feature extraction module, a frame data dual-branch input module, a dual-branch parallel denoising module, and an adaptive gating fusion module, wherein: The noise stationarity feature extraction module extracts noise stationarity features frame by frame from the preprocessed complex spectrum data to obtain the noise stationarity feature extraction results for each frame. The frame data dual-branch input module simultaneously inputs each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch, enabling the two branches to perform parallel noise reduction processing on the same frame of data. The dual-branch parallel noise reduction module controls the steady background noise branch and the non-steady burst noise branch to perform noise reduction processing in parallel, while ensuring that the dimensions of the output features of the two branches remain consistent. The stable background noise branch is denoised using a depthwise separable convolution with a fixed receptive field to obtain the stable branch output features. The non-stationary burst noise branch is denoised using deformable convolution with dynamic sampling offset range to obtain the non-stationary branch output features. The adaptive gating fusion module generates adaptive fusion weights based on the noise stationarity feature extraction results. It then uses a noise feature-driven adaptive gating mechanism to weight and fuse stationary branch output features with non-stationary branch output features to obtain denoised complex spectrum data. A speech synthesis enhancement processing unit, wherein the speech synthesis enhancement processing unit uses deep learning enhancement technology to perform speech synthesis enhancement processing on the noise-reduced complex spectrum data to obtain synthesized enhanced complex spectrum data; The speech data output unit performs integrity, amplitude, and format checks on the synthesized and enhanced speech data, and then outputs the final time-domain speech data.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs a branch allocation mechanism guided by a noise stationarity coefficient. It divides noisy speech frame by frame into a stationary background noise branch composed of separable convolutions with a fixed receptive field depth and a non-stationary burst noise branch composed of deformable convolutions with a dynamic receptive field for differentiated noise reduction. Furthermore, it utilizes the same noise stationarity coefficient to drive a gating network to adaptively fuse the outputs of the two branches. This overcomes the shortcomings of existing single network structures that cannot effectively suppress two types of noise with significantly different properties simultaneously, achieving an optimal balance between the noise reduction effects of stationary background noise and non-stationary burst noise within a unified framework. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall method steps of the deep learning-based speech denoising and speech synthesis enhancement method in this invention; Figure 2 This is a schematic diagram of the overall framework of the deep learning-based speech denoising and speech synthesis enhancement method in this invention; The meanings of the labels in the diagram are as follows: 1. Voice data preprocessing unit; 2. Voice noise reduction processing unit; 3. Speech synthesis and enhancement processing unit; 4. Voice data output unit. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, this embodiment provides a deep learning-based method for speech denoising and speech synthesis enhancement, including: S1. Speech data preprocessing: The original noisy speech data is subjected to frame-by-frame windowing and short-time Fourier transform to extract complex spectrum features and complete standardization to obtain preprocessed complex spectrum data. In this embodiment, the process of preprocessing voice data in S1 includes the following steps: S11. Sample and normalize the original noisy speech data to unify the sampling rate and bit depth of the speech data and obtain the original noisy speech data in a standardized format. In this step, the original noisy speech data collected by different acquisition terminals (such as microphones, recording devices, and smart terminals) have significant differences in sampling rate and data bit depth, and cannot be directly used for short-time Fourier transform (STFT). Therefore, the format standardization process is performed first: the sampling rate of all original noisy speech data is uniformly resampled to 16kHz (the standard sampling rate in the field of speech signal processing, which takes into account both speech fidelity and computational efficiency), the data bit depth is uniformly standardized to 16bit, and empty frames and silent redundant segments in the speech data are removed.

[0019] This format normalization process eliminates the problem of inconsistent input data formats caused by different acquisition devices, ensuring the consistency of input data in the subsequent STFT feature extraction stage and avoiding interference from format differences with the feature extraction effect.

[0020] After S11 is completed, the basic formatting of the original noisy speech data is finished, and the core speech feature extraction stage is entered. The physical frequency domain features of the speech are obtained through short-time Fourier transform.

[0021] S12. Perform frame-segmentation, windowing, and short-time Fourier transform on the original noisy speech data in the standardized format to obtain the original complex spectral feature set (including amplitude spectrum and phase spectrum). In this step, for the original noisy speech data in the standardized format, the time-domain speech signal is converted into frequency-domain complex spectrum features through short-time Fourier transform (STFT), which fully preserves the amplitude and phase information of the speech and forms a physical feature representation that is strictly inverse to ISTFT. No additional deep learning convolutional feature extraction is required, thus ensuring the reversible reconstruction of the speech signal from the source.

[0022] Compared to abstract deep learning features, complex spectral features have clear physical meaning and can completely preserve the core acoustic information and noise distribution features of speech, providing a standardized and reproducible feature basis for subsequent noise reduction processing.

[0023] After S12 is completed, there are still some abnormal feature points and irregular dimensions in the original complex spectrum feature set. Further purification and regularization are needed to ensure the effectiveness of subsequent processing.

[0024] S13. Perform outlier removal and dimension normalization on the original complex spectrum feature set to remove invalid feature data and obtain the normalized complex spectrum feature set. This step first performs outlier detection and removal on the original complex spectrum feature set: outliers with feature amplitudes exceeding the normal speech feature range are removed by using a threshold, and invalid feature data caused by acquisition interference and signal distortion are removed to avoid invalid features interfering with the processing effect of subsequent noise reduction models; then feature dimension normalization is performed: the feature data after outlier removal is uniformly normalized into feature vectors of fixed dimensions to adapt to the input requirements of subsequent deep learning noise reduction models and eliminate the computational obstacles caused by inconsistent feature dimensions.

[0025] After outlier removal and dimensionality normalization, a normalized complex spectrum feature set with no redundancy, no distortion, and uniform dimensions is obtained, which improves the effectiveness and usability of the feature data.

[0026] After S13 is completed, the normalized complex spectrum feature set still has dimensional differences between different feature dimensions, which need to be further standardized to eliminate the influence of dimensions.

[0027] S14. Using a deep learning normalization algorithm, the amplitude and phase of the normalized complex spectrum feature set are normalized to eliminate the difference in feature dimensions and obtain preprocessed complex spectrum data.

[0028] Specifically, this step uses the mean-variance normalization algorithm commonly used in the field of deep learning to normalize the amplitude and phase components of the normalized complex spectrum feature set: calculate the mean and variance of each feature dimension, map the feature data to a unified numerical range, eliminate the numerical dimension difference between the amplitude and phase of the complex spectrum, and avoid problems such as non-convergence of subsequent model training and decreased processing accuracy caused by dimensional imbalance.

[0029] After mean normalization, all feature data are evenly distributed and numerically standardized, resulting in preprocessed complex spectrum data, which serves as pure, regular, and standardized input data for deep learning models.

[0030] S2. Speech noise reduction processing: Extract noise stationarity features from the preprocessed complex spectrum data, and simultaneously input each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch for parallel noise reduction processing. The stationary background noise branch uses a depthwise separable convolution with a fixed receptive field to denoise the complex spectrum features, while the non-stationary burst noise branch uses a deformable convolution with a dynamic sampling offset range to denoise the complex spectrum features. By using an adaptive gating mechanism driven by noise features, adaptive weights are generated based on noise stationarity features and the outputs of the two branches are weighted and fused to obtain denoised complex spectrum data. In this embodiment, the speech noise reduction process in step S2 includes the following steps: S21. Perform frame-by-frame noise stationarity feature extraction on the preprocessed complex spectrum data to obtain the noise stationarity feature extraction result for each frame; the frame-by-frame noise stationarity feature extraction process in S21 includes the following steps: S21.1 Extract the preprocessed complex spectrum data frame by frame to obtain the first frame. Frame complex spectrum data; In this step, the preprocessed complex spectrum data has already undergone frame segmentation processing via S1's STFT, eliminating the need for further frame segmentation and windowing. The data can be directly extracted in time sequence. Frame-by-frame spectral data, preserving the integrity of the original time sequence and frequency domain structure.

[0031] S21.2, calculate the first one respectively. Short-time energy, short-time zero-crossing rate, and spectral flatness of frame complex spectrum data; In this step, based on the complex spectrum amplitude spectrum, three core features are simultaneously calculated from both the time and frequency domains to comprehensively characterize the first... The noise stationarity characteristics of frame speech are calculated as follows: Short-time energy calculation: Short-time energy is used to characterize the first The amplitude intensity of a frame of speech signal is a fundamental time-domain feature that distinguishes speech from noise. It is calculated based on the complex spectrum amplitude spectrum, and the formula is as follows: ; in: Indicates the first Short-time energy of frame complex spectrum data; Indicates the first Frame number Complex spectral amplitude at each frequency point; This represents the total number of frequency points in a single frame of complex spectrum.

[0032] Short-time zero-crossing rate calculation: Short-time zero-crossing rate is used to characterize the first The frequency at which the waveform of a frame of speech signal crosses zero reflects the proportion of high-frequency components in the signal. Based on the time-domain waveform reconstruction using complex spectrum, the calculation formula is as follows: ; in: Indicates the first Short-time zero-crossing rate of frame complex spectrum data; The sign function is defined as follows: it takes the value 1 when the signal amplitude is greater than 0, 0 when it is equal to 0, and -1 when it is less than 0. Indicates the first The time-domain waveform after reconstructing the frame speech signal is in the first... The amplitude of each sampling point.

[0033] Spectral flatness calculation: Spectral flatness is used to characterize the first The stationarity of the frequency domain distribution of a frame-by-frame speech signal is a core frequency domain feature for determining whether noise is stationary. This is directly based on the... The amplitude spectrum of the frame complex spectrum is calculated using the following formula: ; in: Indicates the first Spectral flatness of frame complex spectrum data; Indicates the first Frame complex spectrum data Spectral amplitude at each frequency point; Indicates frequency point index; This represents a very small constant to prevent singular values ​​from occurring in logarithmic operations; This represents the total number of frequency points in a single frame of complex spectrum.

[0034] S21.3 Perform maximum-minimum normalization on the short-time energy and short-time zero-crossing rate in the [0,1] interval; In this step, the short-time energy and short-time zero-crossing rate are subjected to max-min normalization in the [0,1] interval to unify the dimensions and numerical range, and the spectral flatness is... Since the natural value is in the interval [0,1], no normalization is required. The calculation formula is: ; ; in: This represents the normalized short-time energy. , These represent the minimum and maximum values ​​of the short-time energy of the global speech frame, respectively. This represents the normalized short-time zero-crossing rate; , These represent the minimum and maximum values ​​of the short-time zero-crossing rate of the global speech frame, respectively.

[0035] S21.4. The normalized short-time energy, the normalized short-time zero-crossing rate, and the spectral flatness are weighted and summed to obtain the... Noise stationarity coefficient of frame complex spectrum data The sum of the weighting coefficients is 1; In this step, the three features are weighted and fused to obtain a single quantitative index, which can intuitively characterize the noise stability of a single frame of speech. The calculation formula is as follows: ; in: Indicates the first Noise stationarity coefficient of frame complex spectrum data; , , These represent the weighting coefficients for normalized short-time energy, normalized short-time zero-crossing rate, and spectral flatness, respectively, with the sum of their values ​​being 1.

[0036] Noise stability coefficient The value range is [0,1]. The larger the value, the stronger the noise stability of the current frame of speech.

[0037] S21.5, Noise stationarity coefficients of all frames The noise stationarity features are extracted by sequentially concatenating the speech frames. Specifically, the noise stationarity coefficients for each frame are extracted according to their temporal order. By sequentially concatenating these features, a one-dimensional feature sequence that is perfectly aligned with the time sequence of the original preprocessed complex spectrum data is generated. This sequence is the noise stationarity feature extraction result and can be directly used for subsequent fusion weight calculation.

[0038] S22. The preprocessed complex spectrum data of each frame is simultaneously input into the stationary background noise branch and the non-stationary burst noise branch, allowing the two branches to perform parallel noise reduction processing on the same frame of data; in S22, based on the first... The calculation of the frame noise stationarity coefficients for the dual-branch fusion weights includes the following steps: S22.1, Extracting the noise stationarity feature from the extraction results. Frame noise stationarity coefficient ; In this step, based on the noise stationarity coefficient sequence obtained in step S21, which is perfectly time-aligned with the preprocessed complex spectrum data, the noise stationarity coefficients are extracted frame by frame. Noise stationarity coefficient corresponding to the frame .

[0039] S22.2, Based on the aforementioned noise stability coefficient Determine the first The weights for the stationary and non-stationary branches corresponding to a frame are fused together, and the sum of the two fused weights is 1.

[0040] In this step, when the first Frame noise stationarity coefficient When the noise level is greater than or equal to the preset stationarity threshold, it indicates that the noise in the current frame's complex spectrum data is stationary background noise (such as air conditioner operation noise, fan noise, etc.). This type of noise has a long-term stable energy distribution and its spectral characteristics change gradually without significant abrupt changes. In this case, the weighting of the stationary branch fusion is higher. When the... Frame noise stationarity coefficient When the noise level is less than the preset stationarity threshold, it indicates that the noise in the current frame complex spectrum data is non-stationary burst noise (such as knocking sound, sudden environmental interference sound, etc.). This type of noise has strong abrupt changes in energy distribution, large fluctuations in spectral characteristics, and no fixed distribution pattern. At this time, the weight of non-stationary branch fusion is higher.

[0041] The weight calculation rules in this step are as follows: , It can accurately match the noise type and noise reduction requirements of the current frame, providing a weight basis for subsequent adaptive gating fusion.

[0042] This weight calculation method allows for dynamic adjustment of the fusion ratio of the dual-branch outputs based on noise stationarity characteristics. This avoids speech distortion caused by over-processing stationary noise and addresses the insufficient suppression of sudden noise by traditional fixed convolution, laying the foundation for efficient and accurate execution of subsequent dual-branch parallel noise reduction. Furthermore, the frame-by-frame weight calculation is perfectly aligned with the timing of the preprocessed complex spectrum data, ensuring the temporal continuity of the speech signal and preventing frame order misalignment.

[0043] S23. The stationary background noise branch and the non-stationary burst noise branch perform denoising processing in parallel, and the dimensions of the complex spectrum features output by the two branches are consistent. The stationary background noise branch uses a depthwise separable convolution with a fixed receptive field to denoise the complex spectrum features, resulting in the stationary branch output complex spectrum features. The non-stationary burst noise branch uses a deformable convolution with a dynamic sampling offset range to denoise the complex spectrum features, resulting in the non-stationary branch output complex spectrum features. On the one hand, the first The frame preprocessing complex spectrum data is synchronously input into the stationary background noise branch. The corresponding noise is stationary background noise with a fixed spectrum and temporal distribution, so there is no need to dynamically adjust the convolution parameters. Therefore, a one-dimensional depthwise separable convolution with a fixed receptive field that adapts to the frequency domain characteristics of the complex spectrum is used to achieve lightweight and efficient noise reduction, including the following steps: S23.11, the first Frame preprocessing complex spectral data is input into a depth-separable convolution with a fixed receptive field; In this step, the input is the first one assigned to S22. Frame preprocessing complex spectrum data This data is the normalized complex spectrum feature data output by S1, with dimensions of [missing information]. ( The number of input feature channels, (This refers to the feature length of a single frame). This branch uses a one-dimensional depthwise separable convolution with a fixed receptive field, and the receptive field size is... For a preset fixed value (in this embodiment, it is taken as...) The convolution kernel parameters remain unchanged during inference, adapting to the frequency domain distribution characteristics of stationary background noise, and only adapting to the fixed distribution characteristics of stationary background noise.

[0044] S23.12. Perform a depthwise convolution operation, performing convolution calculations on each input channel of the complex spectral features separately; In this step, depthwise convolution is channel-independent convolution, performing convolution calculations separately for each channel of the input features, without fusing information across channels. The calculation formula is as follows: ; in: Indicates the first Frame, First The depthwise convolution output features of each channel; Indicates the first Frame, First Input preprocessing complex spectral characteristics of each channel; Indicates the first Fixed-depth convolutional kernel weights for each channel; Indicates the fixed receptive field size; Represents the time-domain index of the feature.

[0045] This step extracts only the stationary noise features within a single channel, ensuring noise reduction efficiency without disrupting the frequency domain structure of the complex spectrum.

[0046] S23.13 Perform pointwise convolution operation to perform channel fusion on the output of depthwise convolution, and maintain the integrity of the complex spectral feature structure; In this step, the pointwise convolution is a 1×1 one-dimensional convolution. Its core function is cross-channel feature fusion and output dimension regularization, ensuring that the output complex spectral features are completely aligned with the frequency domain dimension of the non-stationary branch. The calculation formula is as follows: ; in: This represents the fused features output by pointwise convolution; Indicates the pointwise convolution kernel weights; This indicates the number of input feature channels.

[0047] This step fixes the number of output feature channels to [value]. (This embodiment) This dimension is forcibly aligned with the output dimension of the non-stationary branch to meet the requirements of subsequent fusion.

[0048] S23.14. Output the processed complex spectrum feature data as the first... The smooth branching output characteristics of the frame.

[0049] In this step, after depthwise convolution and pointwise convolution, the stable background noise is completely suppressed, and the output is the first... Frame smooth branch output characteristics Its dimensions are It has the same feature dimensions as the output of non-stationary branches and fully preserves the physical structure of the complex spectrum, providing compliant input for subsequent fusion.

[0050] On the other hand, the The frame preprocessing complex spectrum data is synchronously input into the non-stationary burst noise branch. This noise is non-stationary burst noise with irregular frequency domain distribution and random location. Therefore, a deformable convolution with a dynamic sampling offset range adapted to the characteristics of the complex spectrum frequency domain is employed. Dynamically adjusting convolution parameters to accurately capture and suppress sudden noise includes the following steps: S23.21, according to the... Frame noise stationarity coefficient Determine the first Normalized intensity levels of frame nonstationary burst noise; Specifically, in combination With normalized short-time energy The calculation formula is as follows: ; in: Indicates the first The normalized intensity level of burst noise in frames, with a value range of [0,1]. This represents the normalized short-time energy in S21.3.

[0051] pass Quantify the burstiness of noise ( The smaller, The larger the size, the higher the degree of suddenness (combined with...) The reflected energy intensity, when added together and then divided by 2, yields a normalized intensity level that integrates the burst characteristics and energy level; because , The sum of the two is in the range [0,2]. Dividing by 2... Normalized to the [0,1] interval, it can be directly used as the weight for subsequent parameter adjustment without additional normalization processing.

[0052] S23.22, Based on noise stationarity coefficient The effective sampling coverage of deformable convolution is dynamically adjusted by adjusting the amplitude range of the sampling offset without changing the fixed receptive field size of the convolution kernel. In this step, the first Maximum allowed frame offset From the noise stability coefficient The decision is made using the following formula: ; in, For the first The maximum allowable offset of a frame deformable convolution, in units of sampling points; To preset the minimum allowable offset, this embodiment sets the value to 1, with the unit being sampling points; The maximum allowable offset is preset, and in this embodiment, the value is 5, with the unit being sampling points; The smaller the value, the larger the maximum allowable offset, and the wider the sampling point distribution range of deformable convolution, thus adapting to non-stationary burst noise with a larger time span. Actual offset Within this maximum permissible range, the offset prediction layer of deformable convolution is further refined and adjusted.

[0053] S23.23, Based on the noise stability coefficient The normalized intensity level of non-stationary burst noise is used to dynamically adjust the kernel offset of deformable convolution, where the noise stationarity coefficient is... The smaller the value and the higher the normalization intensity level, the greater the convolution kernel offset. In this step, to ensure that the sampling position of the deformable convolution is accurately aligned with the center of the burst noise, the kernel offset needs to be dynamically adjusted according to the burst level and intensity of the noise. The stronger the non-stationarity of the noise ( Smaller), higher strength ( The larger the value, the larger the offset, thus enhancing the sampling weight for noisy regions. The calculation formula is as follows: ; in: Indicates the first Dynamic kernel offset of frame deformable convolution; This is the offset scaling factor, with units of sample points / dimensionless. Its function is to map the dimensionless noise feature weights to the sample point offset of the deformable convolution (in this embodiment, it is taken as...). (This is the preferred value for adapting the signal frame length and convolution kernel configuration).

[0054] S23.24. Apply the adjusted deformable convolution to the first... The frame preprocessing complex spectrum data is convolved to extract and suppress non-stationary burst noise features; In this step, based on dynamic adjustment Perform deformable convolution calculation, the calculation formula is: ; This represents the output features of deformable convolution; Indicates the deformable convolution kernel weights; Indicates the first Frame input preprocessing complex spectral characteristics.

[0055] This step extracts burst noise features by dynamically sampling locations, and performs noise suppression through feature mapping, retaining only valid speech information.

[0056] S23.25. Output the processed complex spectrum feature data as the first... Non-stationary branching output characteristics of frames.

[0057] In this step, after deformable convolution denoising, the output is the first... Frame non-stationary branch output characteristics Its dimension is forcibly set to , with stable branch output characteristics The dimensions are completely consistent, and the physical structure of the complex spectrum is fully preserved, providing compliant input for subsequent adaptive gating fusion.

[0058] S24. Based on the noise stationarity feature extraction results, adaptive fusion weights are generated. The stationary branch output features and non-stationary branch output features are weighted and fused using a noise feature-driven adaptive gating mechanism to obtain denoised complex spectrum data. The adaptive gating fusion process in S24 includes the following steps: S24.1, according to the first Frame noise stationarity coefficient Calculate the first Frame smooth branch fusion weights Weights for merging non-stationary branches ,in , In this step, the weights are directly derived from the noise stationarity coefficient. The generation process accurately matches the noise type and denoising requirements of the current frame. The two sets of weights are calculated as follows: ; In the formula, For the first The steady branch fusion weights corresponding to the frames For the first The non-stationary branch fusion weights corresponding to the frames. The first one obtained in S21 Frame noise stationarity coefficient. Since the sum of the two weights is always 1, it ensures that the fused feature energy is stable and reasonably distributed, and there will be no signal distortion or amplitude abnormality.

[0059] S24.2, An adaptive gating mechanism driven by noise characteristics is used to fuse weights according to the stable branch. Weights for merging non-stationary branches For the same first The smooth branch output characteristics of the frame and the first The non-stationary branch output features of the frame are summed element-wise using a weighted summation. In this step, since the output features of the two branches have been adjusted to have exactly the same dimension in S23, the two sets of features can be directly fused element-wise according to the same frequency position. The fusion calculation formula is as follows: ; In the formula, For the first Denoising characteristics after frame fusion For smooth branch output characteristics, This is a non-stationary branch output characteristic.

[0060] The above fusion method is driven by noise characteristics and is a typical implementation of adaptive gating mechanism: frames with a high proportion of stable noise will retain more of the noise reduction effect of the stable branch; frames with a high proportion of burst noise will adopt more of the processing results of the non-stable branch, which takes into account both noise reduction accuracy and ensures the naturalness and integrity of the speech.

[0061] S24.3. The weighted summation results of all frames are concatenated in time sequence to obtain the denoised complex spectrum data.

[0062] In this step, the denoised complex spectrum features obtained by frame-by-frame fusion are spliced ​​together in chronological order to generate complete denoised complex spectrum data that is perfectly aligned with the original preprocessed complex spectrum data in terms of time sequence. All frequency domain features and temporal structure are preserved, and no inverse short-time Fourier transform operation is performed, so as to provide compliant frequency domain input for the subsequent S3 speech synthesis enhancement processing.

[0063] S3. Speech Synthesis Enhancement Processing: Deep learning enhancement technology is used to perform speech synthesis enhancement processing on the denoised complex spectrum data to obtain synthesized enhanced complex spectrum data. In this embodiment, the speech synthesis enhancement process in step S3 includes the following steps: S3.1 Perform frame structure calibration on the denoised complex spectrum data to obtain the normalized denoised complex spectrum data; Specifically, the denoised complex spectrum output by S24 is already the frequency domain data of the standard frame segmentation in S1. Each frame has the same frequency point dimension (determined by the number of FFT points), and the inter-frame timing is determined by the frame shift of S1. This step does not require further frame segmentation and windowing; it only performs: Dimension verification: Confirm that the number of frequency points in each frame is consistent with the number of FFT points in S1; Timing continuity check: Confirm that the frame sequence is free of missing or duplicate frames, and that the timestamps are aligned with the input. Optional inter-frame smoothing: A moving average of the complex spectral amplitudes of adjacent frames (with a window length matched to the S1 frame shift) is applied to eliminate inter-frame abrupt changes introduced by noise reduction.

[0064] The above calibration ensures the consistency of the frame structure for subsequent acoustic feature extraction.

[0065] S3.2 Extract the acoustic features of the framed, denoised complex spectrum data, including fundamental frequency features, formant features and Mel spectrum features; In this step, core acoustic features of the speech are extracted based on the denoised complex spectrum data to comprehensively characterize the speech's tone quality, timbre, and prosodic attributes. These three types of features are standard features in the field of speech processing, as detailed below: Fundamental frequency features: Based on the extraction of low-frequency energy distribution in complex spectrum, it reflects the fundamental frequency of speech, characterizes the pitch and prosodic characteristics of speech, and is the core feature to ensure the naturalness of speech. Formant features: Based on the extraction of frequency domain peak distribution of complex spectrum, it reflects the frequency response characteristics of vocal tract, determines the timbre attributes of speech, and is used to repair and unify the timbre performance of speech after noise reduction. Mel spectral features: Based on the frequency domain data of the complex spectrum, they are obtained through Mel filter bank transformation. They closely match human auditory perception and are the core feature carrier for speech enhancement and sound quality optimization.

[0066] By jointly extracting the above three types of features, the effective frequency domain information of the denoised speech can be fully preserved, while accurately locating the sound quality defects that need to be enhanced and optimized.

[0067] S3.3. A deep learning enhancement model is used to process the extracted acoustic features and complex spectral features. The deep learning enhancement model uses the first... Frame noise stationarity coefficient As a conditional input, the enhancement intensity or feature fusion weights are dynamically adjusted. In this step, a deep learning speech enhancement model combining one-dimensional convolutional neural networks (1D-CNN) and gated recurrent units (GRU) is employed. This model utilizes the fundamental frequency, formants, and Mel-spectral feature vectors extracted in step S3.2, as well as the noise stationarity coefficients for each frame obtained in step S2. As input, the specific structure and processing logic of the model are as follows: The deep learning speech enhancement model consists of an input layer, a feature encoding layer, a conditional modulation layer, a temporal modeling layer, a feature reconstruction layer, and an output layer.

[0068] Input layer: Concatenates fundamental frequency, formants, and Mel-frequency spectral features along the channel dimension to form a fused acoustic feature vector; simultaneously receives the noise stationarity coefficient of each frame. As a conditional control signal.

[0069] Feature encoding layer: Composed of 3 layers of one-dimensional convolutional network, used to extract and encode local features of fused acoustic features and output intermediate feature maps.

[0070] Conditional modulation layer: This modulates the noise level coefficient. The encoded feature map is mapped to a modulation vector with the same number of channels as the intermediate feature map through a fully connected layer. Then, the encoded feature map is modulated through a feature affine transformation (scaling and offset), so that the intensity of the enhancement process varies. Adaptive change - when When the amplitude is large (close to 1, dominated by stationary noise), the modulation layer weakens the enhancement amplitude, preserving more of the original acoustic characteristics; when When the noise level is low (close to 0, dominated by non-stationary burst noise), the modulation layer enhances the amplitude, resulting in more aggressive suppression of residual noise.

[0071] Temporal modeling layer: A two-layer bidirectional GRU network is used to perform temporal dependency modeling on the conditionally modulated feature sequence and output features with contextual information.

[0072] Feature Reconstruction Layer: Through 1×1 pointwise convolution, the output features of the temporal modeling layer are fused and dimensionally normalized to output enhanced features with the same dimension as the input acoustic features.

[0073] The model is trained in a supervised manner. The training dataset consists of paired clean speech features and noisy speech features. During training, the model is trained using... As an auxiliary input condition, the mean squared error (MSE) loss function is selected, with the training objective being to minimize the error between the model output and the clean speech features. After processing by this model, the output acoustic features have been... Differentiation enhancements were implemented.

[0074] S3.4. Post-process the processed complex spectrum feature data to remove outliers and residual noise. In this step, a combination of sliding window threshold judgment and one-dimensional mean filtering is used. First, the numerical distribution of the three types of features, namely fundamental frequency, formants, and Mel spectrum, is traversed and detected. Abnormal points that exceed the normal range of speech feature values ​​are removed and the mean of neighboring features is used to fill in the gaps, so as to avoid distortion of subsequent speech waveforms due to feature abnormalities. Then, a lightweight one-dimensional mean filter is used to smooth the feature sequence, remove the weak noise and high-frequency interference remaining in the features, and further purify the feature data.

[0075] After post-processing, the numerical distribution of acoustic features is regular, without anomalies or redundant residues, providing clean basic data for subsequent inverse transform to restore high-quality speech signals.

[0076] S3.5. Perform inverse short-time Fourier transform and overlap summation on the post-processed synthesized enhanced complex spectrum data to obtain synthesized enhanced speech data.

[0077] Specifically, a signal transformation operation that is strictly inverse of the Short-Time Fourier Transform (STFT) in S1 is used to restore the optimized frequency-domain acoustic features into a continuous time-domain speech signal, achieving complete closed-loop reconstruction of the speech signal. The specific implementation steps are as follows: The first step is the Mel spectrum inverse transform: the post-processed optimized Mel spectrum features are converted into a linear frequency domain amplitude spectrum through the pseudo-inverse matrix of the Mel filter bank (or by using an iterative algorithm).

[0078] The second step is fine-tuning of the spectrum: combining the calibrated fundamental frequency characteristics and formant characteristics, the linear frequency domain amplitude spectrum is locally adjusted so that its fundamental structure and formant distribution match the enhanced acoustic characteristics, thus obtaining the final reconstructed frequency domain amplitude spectrum.

[0079] The third step is phase recovery and complex spectrum construction: The original phase spectrum is extracted from the denoised complex spectrum data output by S2 (since the denoising process does not change the phase, it can be directly reused); if the phase is not retained for storage or transmission considerations, phase estimation algorithms such as Griffin-Lim are used to iteratively estimate the phase from the reconstructed amplitude spectrum. The reconstructed amplitude spectrum and phase spectrum are combined to form a complete complex spectrum.

[0080] The fourth step is inverse short-time Fourier transform and overlap addition: perform inverse short-time Fourier transform (IFFT) on the complex spectrum of each frame to obtain a single-frame time-domain speech signal; finally, perform overlap addition (OLA) on all single-frame time-domain signals according to the same frame length, frame shift and window function parameters as when framing and windowing in S1, to eliminate the discontinuity between frames, and splice the discrete single-frame signals into a coherent and smooth continuous time-domain speech signal, finally obtaining the synthesized enhanced time-domain speech data.

[0081] This time-domain speech data, based on noise suppression, achieves a comprehensive improvement in clarity, naturalness, and timbre consistency, and can be directly used in subsequent speech data output stages.

[0082] S4. Speech Data Output: After verifying the synthesized and enhanced speech data, output the final time-domain speech data.

[0083] In this step, multi-dimensional verification is performed on the synthesized enhanced time-domain speech data, including data integrity verification to confirm that there are no dropped frames, no interruptions, and the duration is completely aligned with the original noisy speech; signal amplitude verification to detect that the speech signal has no overload distortion, no abnormal silence segments, and the amplitude distribution conforms to the standard speech signal range; and format standardization verification to ensure that the speech sampling rate and data bit depth are completely consistent with the format in the S1 speech data preprocessing stage.

[0084] Verified time-domain speech data is directly output as the final result; if there are minor issues such as small amplitude deviations, the speech signal is linearly calibrated before being output.

[0085] After this process, the final output time-domain speech data has the characteristics of being free from noise interference, having high definition, high naturalness, and uniform timbre, thus fully realizing the whole-process optimization process from the original noisy speech to high-quality speech.

[0086] like Figure 2 As shown, this embodiment also provides a deep learning-based speech denoising and speech synthesis enhancement system. The computer program for this system, when running, performs the steps of the aforementioned deep learning-based speech denoising and speech synthesis enhancement method, including: Speech data preprocessing unit 1 performs frame-segmentation, windowing, and short-time Fourier transform on the original noisy speech data, extracts complex spectrum features, and completes standardization processing to obtain preprocessed complex spectrum data. Speech denoising processing unit 2 performs denoising processing on the preprocessed complex spectrum data to obtain denoised complex spectrum data. Speech denoising processing unit 2 includes a noise stationarity feature extraction module, a frame data dual-branch input module, a dual-branch parallel denoising module, and an adaptive gating fusion module, wherein: The noise stationarity feature extraction module extracts noise stationarity features frame by frame from the preprocessed complex spectrum data, and obtains the noise stationarity feature extraction results for each frame. The frame data dual-branch input module simultaneously inputs each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch, enabling the two branches to perform parallel noise reduction processing on the same frame of data. The dual-branch parallel noise reduction module controls the steady-state background noise branch and the non-stationary burst noise branch to perform noise reduction processing in parallel, while ensuring that the dimensions of the output features of the two branches remain consistent. The stable background noise branch is denoised using depthwise separable convolution with a fixed receptive field to obtain the stable branch output features; The non-stationary burst noise branch is denoised using deformable convolution with dynamic sampling offset range to obtain the non-stationary branch output features; The adaptive gating fusion module generates adaptive fusion weights based on the noise stationarity feature extraction results. It then uses a noise feature-driven adaptive gating mechanism to weight and fuse stationary branch output features with non-stationary branch output features to obtain denoised complex spectrum data. The speech synthesis and enhancement processing unit 3 uses deep learning enhancement technology to perform speech synthesis and enhancement processing on the noise-reduced complex spectrum data to obtain synthesized and enhanced complex spectrum data. The speech data output unit 4 performs integrity, amplitude, and format checks on the synthesized and enhanced speech data, and then outputs the final time-domain speech data.

[0087] Those skilled in the art will understand that the process of implementing all or part of the steps of the above embodiments can be carried out by hardware or by a program instructing the relevant hardware.

[0088] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A deep learning-based method for speech denoising and speech synthesis enhancement, characterized in that, include: S1. Speech data preprocessing: The original noisy speech data is subjected to frame-by-frame windowing and short-time Fourier transform to extract complex spectrum features and complete standardization to obtain preprocessed complex spectrum data. S2. Speech noise reduction processing: Extract noise stationarity features from the preprocessed complex spectrum data, and simultaneously input each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch for parallel noise reduction processing. The stationary background noise branch uses a depthwise separable convolution with a fixed receptive field to denoise the complex spectrum features, while the non-stationary burst noise branch uses a deformable convolution with a dynamic sampling offset range to denoise the complex spectrum features. By using an adaptive gating mechanism driven by noise features, adaptive weights are generated based on noise stationarity features and the outputs of the two branches are weighted and fused to obtain denoised complex spectrum data. S3. Speech Synthesis Enhancement Processing: Deep learning enhancement technology is used to perform speech synthesis enhancement processing on the denoised complex spectrum data to obtain synthesized enhanced complex spectrum data. S4. Speech Data Output: After verifying the synthesized and enhanced speech data, output the final time-domain speech data.

2. The deep learning based speech denoising and speech synthesis enhancement method according to claim 1, characterized in that, In step S1, the process of preprocessing voice data includes the following steps: S11. Sample and normalize the original noisy speech data to unify the sampling rate and bit depth of the speech data and obtain the original noisy speech data in a standardized format. S12. Perform frame-splitting, windowing, and short-time Fourier transform on the original noisy speech data in the standardized format to obtain the original complex spectrum feature set; S13. Perform outlier removal and dimension normalization on the original complex spectrum feature set to remove invalid feature data and obtain the normalized complex spectrum feature set. S14. Using a deep learning normalization algorithm, the amplitude and phase of the normalized complex spectrum feature set are normalized to eliminate the difference in feature dimensions and obtain preprocessed complex spectrum data.

3. The deep learning-based speech denoising and speech synthesis enhancement method according to claim 1, characterized in that, In S2, the process of performing speech noise reduction includes the following steps: S21. Perform frame-by-frame noise stationarity feature extraction on the preprocessed complex spectrum data to obtain the noise stationarity feature extraction results for each frame. S22. Input each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch simultaneously, so that the two branches can perform noise reduction processing on the same frame of data in parallel. S23. The stationary background noise branch and the non-stationary burst noise branch perform noise reduction processing in parallel, and the dimensions of the complex spectrum features output by the two branches are consistent. The stationary background noise branch uses a depthwise separable convolution with a fixed receptive field to perform noise reduction processing on the complex spectrum features, and the non-stationary burst noise branch uses a deformable convolution with a dynamic sampling offset range to perform noise reduction processing on the complex spectrum features, and the non-stationary branch output complex spectrum features. S24. Based on the noise stationarity feature extraction results, an adaptive fusion weight is generated. The stationary branch output features and the non-stationary branch output features are weighted and fused through an adaptive gating mechanism driven by noise features to obtain denoised complex spectrum data.

4. The deep learning based speech denoising and speech synthesis enhancement method according to claim 3, characterized in that, In step S21, the process of extracting frame-by-frame noise stationarity features includes the following steps: S21.1 Extract the preprocessed complex spectrum data frame by frame to obtain the first frame. Frame complex spectrum data; S21.2, calculate the first one respectively. Short-time energy, short-time zero-crossing rate, and spectral flatness of frame complex spectrum data; S21.3 Perform maximum-minimum normalization on the short-time energy and short-time zero-crossing rate in the [0,1] interval; S21.

4. The normalized short-time energy, the normalized short-time zero-crossing rate, and the spectral flatness are weighted and summed to obtain the... Noise stationarity coefficient of frame complex spectrum data The sum of the weighting coefficients is 1. S21.5, Noise stationarity coefficients of all frames By performing temporal splicing in chronological order, the noise stationarity feature extraction results are obtained.

5. The deep learning-based speech denoising and speech synthesis enhancement method according to claim 3, characterized in that, In S22, based on the first The calculation of the frame noise stationarity coefficients for the dual-branch fusion weights includes the following steps: S22.1, Extracting the noise stationarity feature from the extraction results. Frame noise stationarity coefficient ; S22.2, Based on the aforementioned noise stability coefficient Determine the first The weights for the stationary and non-stationary branches corresponding to a frame are fused together, and the sum of the two fused weights is 1.

6. The deep learning-based speech denoising and speech synthesis enhancement method according to claim 3, characterized in that, In step S23, the process of performing smooth background noise branch denoising includes the following steps: S23.11, the first Frame preprocessing complex spectral data is input into a depth-separable convolution with a fixed receptive field; S23.

12. Perform a depthwise convolution operation to perform convolution calculations on each input channel of the complex spectral features; S23.

13. Perform pointwise convolution operation to fuse the channels of the output of depthwise convolution and maintain the integrity of the complex spectral feature structure. S23.14, output the processed complex spectrum feature data as the stationary branch output feature of the frame S23.14, output the processed complex spectrum feature data as the stationary branch output feature of the frame 7. The deep learning-based speech denoising and speech synthesis enhancement method according to claim 3, characterized in that, In step S23, the process of performing non-stationary burst noise branch denoising includes the following steps: S23.21, according to the... Frame noise stationarity coefficient Determine the first Normalized intensity levels of frame nonstationary burst noise; S23.22, Based on noise stationarity coefficient The effective sampling coverage of deformable convolution can be dynamically adjusted by adjusting the range of sampling offset without changing the fixed receptive field size of the convolution kernel. S23.23, Based on the noise stability coefficient The normalized intensity level of non-stationary burst noise is used to dynamically adjust the kernel offset of deformable convolution, where the noise stationarity coefficient is... The smaller the value and the higher the normalization intensity level, the greater the convolution kernel offset. S23.

24. Apply the adjusted deformable convolution to the first... The frame preprocessed complex spectrum data is convolved to extract and suppress non-stationary burst noise features; S23.

25. Output the processed complex spectrum feature data as the first... Non-stationary branching output characteristics of frames.

8. The deep learning-based speech denoising and speech synthesis enhancement method according to claim 3, characterized in that, In S24, the adaptive gating fusion process includes the following steps: S24.1, according to the first Frame noise stationarity coefficient Calculate the first Frame smooth branch fusion weights Weights for merging non-stationary branches ,in , ; S24.2, An adaptive gating mechanism driven by noise characteristics is used to fuse weights according to the stable branch. Weights for merging non-stationary branches For the same first The smooth branch output characteristics of the frame and the first The non-stationary branch output features of the frame are summed element-wise using a weighted summation. S24.

3. The weighted summation results of all frames are concatenated in time sequence to obtain the denoised complex spectrum data.

9. The deep learning-based speech denoising and speech synthesis enhancement method according to claim 1, characterized in that, In S3, the process of speech synthesis enhancement includes the following steps: S3.1 Perform frame structure calibration on the denoised complex spectrum data to obtain the normalized denoised complex spectrum data; S3.2 Extract the acoustic features of the framed, denoised complex spectrum data, including fundamental frequency features, formant features and Mel spectrum features; S3.

3. A deep learning enhancement model is used to process the extracted acoustic features and complex spectral features. The deep learning enhancement model uses the [missing information - likely a specific model or method] as the [missing information - likely a specific model or method]. Frame noise stationarity coefficient As a conditional input, the enhancement intensity or feature fusion weights are dynamically adjusted. S3.

4. Post-process the processed complex spectrum feature data to remove outliers and residual noise. S3.5 Perform inverse short-time Fourier transform and overlap summation on the post-processed synthesized enhanced complex spectrum data to obtain synthesized enhanced speech data.

10. A deep learning-based speech denoising and speech synthesis enhancement system, characterized in that, When the computer program of the deep learning-based speech denoising and speech synthesis enhancement system is executed, it performs the steps of the deep learning-based speech denoising and speech synthesis enhancement method according to any one of claims 1-9, including: The speech data preprocessing unit (1) performs frame-by-frame windowing and short-time Fourier transform on the original noisy speech data, extracts complex spectrum features and completes standardization processing to obtain preprocessed complex spectrum data. The speech denoising processing unit (2) performs denoising processing on the preprocessed complex spectrum data to obtain denoised complex spectrum data. The speech denoising processing unit (2) includes a noise stationarity feature extraction module, a frame data dual-branch input module, a dual-branch parallel denoising module, and an adaptive gating fusion module, wherein: The noise stationarity feature extraction module extracts noise stationarity features frame by frame from the preprocessed complex spectrum data to obtain the noise stationarity feature extraction results for each frame. The frame data dual-branch input module simultaneously inputs each frame of preprocessed complex spectrum data into the stationary background noise branch and the non-stationary burst noise branch, enabling the two branches to perform parallel noise reduction processing on the same frame of data. The dual-branch parallel noise reduction module controls the steady background noise branch and the non-steady burst noise branch to perform noise reduction processing in parallel, while ensuring that the dimensions of the output features of the two branches remain consistent. The stable background noise branch is denoised using a depthwise separable convolution with a fixed receptive field to obtain the stable branch output features. The non-stationary burst noise branch is denoised using deformable convolution with dynamic sampling offset range to obtain the non-stationary branch output features. The adaptive gating fusion module generates adaptive fusion weights based on the noise stationarity feature extraction results. It then uses a noise feature-driven adaptive gating mechanism to weight and fuse stationary branch output features with non-stationary branch output features to obtain denoised complex spectrum data. The speech synthesis enhancement processing unit (3) uses deep learning enhancement technology to perform speech synthesis enhancement processing on the noise-reduced complex spectrum data to obtain synthesized enhanced complex spectrum data; The speech data output unit (4) performs integrity, amplitude and format verification on the synthesized enhanced speech data and outputs the final time-domain speech data.

Images