Voice enhancement method, electronic device, and storage medium

Abstract

This invention discloses a speech enhancement method, an electronic device, and a storage medium. The speech enhancement method is used in a speech enhancement system, which includes an encoder module, a dual-channel module, a decoder module, and a merging module. The decoder module includes an excitation decoder, a channel decoder, and a complex spectrum decoder. The method includes: extracting preset features from the original noisy speech signal using the encoder module; modeling the preset features on the frequency axis using the dual-channel module to obtain frequency axis features; modeling the preset features on the time axis using the dual-channel module to obtain time axis features; processing the time axis features using the excitation decoder and channel decoder to obtain a first denoised speech signal; processing the frequency axis features using the complex spectrum decoder to obtain a second denoised speech signal; and merging the first denoised speech signal, the second denoised speech signal, and the original noisy speech signal using the merging module to obtain a final estimated speech signal.

Description

Technical Field

[0001] This invention belongs to the field of speech enhancement technology, and particularly relates to speech enhancement methods, electronic devices, and storage media. Background Technology

[0002] In related technologies, speech enhancement technology processes the speech signal collected by the microphone to reduce noise, thereby improving speech quality and intelligibility. As a front-end processing module of speech application systems, speech enhancement is an important technical means for the implementation of systems such as voice communication, speech recognition, and speaker recognition.

[0003] Speech enhancement technology has a research history of several decades, mainly divided into traditional signal processing-based methods and, in recent years, data-driven neural network methods. Traditional signal processing-based methods have the advantage of requiring no training data and possessing strong algorithmic adaptability, but their noise reduction effect is limited. Neural network noise reduction methods, developed in recent years, have the advantage of good noise reduction performance on closed-set tests, but their generalization ability to noise types not present in the training set is poor. Combining traditional digital signal processing methods and neural network methods can integrate the advantages of both, resulting in a speech enhancement method with better noise reduction performance and stronger generalization ability.

[0004] One data-driven neural network method is the first to use deep neural networks for speech denoising. It employs a fully connected deep neural network structure that processes only the amplitude spectrum of the speech signal. The network input is the amplitude spectrum of the noisy speech signal, and the output is the complex cepstrum of the clean speech signal. Another data-driven neural network method employs a time-frequency mask estimation approach. This type of method uses a neural network to estimate a certain time-frequency mask for the speech signal, such as a binary mask or ratio mask for the amplitude or power spectrum, or a mask in the complex domain. During speech denoising, the denoised speech is obtained by multiplying this mask by the amplitude spectrum, power spectrum, or complex spectrum of the noisy speech. Besides improvements and optimizations in time-frequency masks, neural network denoising methods also explore the structure of neural networks. For example, convolutional neural networks and recurrent neural networks are combined to obtain a convolutional recurrent neural network structure. The convolutional network layers are used to extract the time-frequency information of the signal, and the recurrent network layers are used to model the temporal information of the speech signal. Alternatively, a complex neural network structure can be used to directly process the amplitude spectrum of the speech signal. This network structure processes the real and imaginary parts of the complex number separately and uses the rules of complex number arithmetic for forward propagation.

[0005] Other neural network denoising techniques, which do not employ an end-to-end time-domain approach using Fourier transform, use time-domain speech signals as both input and output. They utilize an encoder-decoder network structure to extract signal features and perform denoising on a specific feature domain. The advantage of this type of method is its ability to fit training data well, but its disadvantage lies in poor generalization ability, thus limiting its use in practical engineering applications. Some existing techniques combine traditional signal processing vocoder structures with neural networks to obtain a neural network homomorphic synthesis speech enhancement method. This type of method effectively utilizes the physical generation model of speech signals, making the modeling method more efficient and the denoising effect more stable. However, this method can only process the amplitude spectrum of the signal, thus limiting the upper limit of the algorithm's performance.

[0006] In the process of developing this application, the inventors discovered that traditional signal processing-based speech enhancement methods do not require training data and have some adaptability to unknown noise, but they rely on noise tracking and estimation methods. The accuracy of the noise tracking and estimation algorithm determines the upper limit of the denoising algorithm's performance, but traditional noise estimation algorithms can generally only track and estimate steady-state noise, performing poorly for non-steady-state noise. On the other hand, neural network-based speech enhancement methods typically do not consider the characteristics of the speech signal itself and rely heavily on a large amount of training data. They can achieve good denoising results in closed-set tests, but their generalization ability on unknown noise types is poor, and there is a risk of over-denoising leading to speech distortion. Summary of the Invention

[0007] This invention provides a speech enhancement method, an electronic device, and a storage medium to at least solve one of the aforementioned technical problems.

[0008] In a first aspect, embodiments of the present invention provide a speech enhancement method for a speech enhancement system, wherein the speech enhancement system includes an encoder module, a dual-channel module, a decoder module, and a merging module, the decoder module including an excitation decoder, a channel decoder, and a complex spectrum decoder, the method comprising: extracting preset features from an original noisy speech signal using the encoder module; modeling the preset features on a frequency axis using the dual-channel module to obtain frequency axis features, and modeling the frequency axis features on a time axis using the dual-channel module to obtain time axis features; processing the time axis features using the excitation decoder and the channel decoder to obtain a first denoised speech signal of a time axis branch; processing the frequency axis features using the complex spectrum decoder to obtain a second denoised speech signal of a frequency axis branch; and merging the first denoised speech signal, the second denoised speech signal, and the original noisy speech signal using the merging module to obtain a final estimated speech signal.

[0009] In a second aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the speech enhancement method of any embodiment of the present invention.

[0010] Thirdly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the speech enhancement method of any embodiment of the present invention.

[0011] In the method of this application embodiment, a dual-path module is used to model and process the time axis and frequency axis respectively. Then, different decoders are used to process the frequency axis features and time axis features respectively, and then they are merged. A single decoder and multi-decoder network structure is used to organically integrate the two into one, thereby achieving speech enhancement with lower model complexity, stronger model interpretability and better noise reduction performance. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a speech enhancement method according to an embodiment of the present invention;

[0014] Figure 2 This is a system block diagram of a speech enhancement method that integrates neural homomorphic synthesis and time-frequency mask estimation according to an embodiment of the present invention;

[0015] Figure 3 A specific structure diagram of a neural network provided in an embodiment of the present invention;

[0016] Figure 4 This is a diagram illustrating a general source filtering model in related technologies;

[0017] Figure 5 This is a diagram illustrating homomorphic analysis of speech in related technologies;

[0018] Figure 6 A block diagram of cepstral analysis and synthesis for speech enhancement in related technologies;

[0019] Figure 7A spectrogram of noise and clean speech provided in an embodiment of the present invention;

[0020] Figure 8 This invention provides a study of neural network structures on the VoiceBand+DEMAND dataset, as an embodiment of the present invention.

[0021] Figure 9 This invention provides a performance comparison with other SOTA methods on the Voicebank+Demand dataset, as provided in one embodiment of the invention.

[0022] Figure 10 This invention provides a performance comparison with other state-of-the-art methods on the DNS Challenge 2020 test set, based on an embodiment of the present invention.

[0023] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please refer to Figure 1 The diagram illustrates a flowchart of an embodiment of the speech enhancement method of this application. The method is used in a speech enhancement system, which includes an encoder module, a dual-channel module, a decoder module, and a merging module. The decoder module includes an excitation decoder, a channel decoder, and a complex spectrum decoder.

[0026] like Figure 1 As shown, in step 101, the encoder module is used to extract preset features from the original noisy speech signal;

[0027] In step 102, the preset features are modeled on the frequency axis using the dual-path module to obtain frequency axis features, and the frequency axis features are modeled on the time axis using the dual-path module to obtain time axis features.

[0028] In step 103, the time axis features are processed using the excitation decoder and the channel decoder to obtain the first denoised speech signal of the time axis branch. In a specific example, the excitation decoder is used to perform signal excitation part modeling processing on the time axis features to obtain excitation features, and the channel decoder is used to perform channel part modeling processing on the time axis features to obtain channel features. The excitation features and the channel features are subjected to preset processing to obtain processed features. The processed features are combined with the phase features of the original noisy speech signal to obtain the first denoised speech signal of the time axis branch. This application is not limited in this respect.

[0029] In step 104, the frequency axis features are processed using the complex spectrum decoder to obtain the second denoised speech signal of the frequency axis branch. In a specific example, the frequency axis features are modeled using the complex spectrum decoder to obtain complex spectrum features. The complex spectrum features are then masked with the spectral features of the original noisy speech signal to obtain the second denoised speech signal of the frequency axis branch. This application does not limit this.

[0030] In step 105, the first denoised speech signal, the second denoised speech signal, and the original noisy speech signal are merged using the merging module to obtain the final estimated speech signal.

[0031] The method in this embodiment models and processes the time axis and frequency axis separately using a dual-path module. Then, different decoders are used to process the frequency axis features and time axis features respectively, and then they are merged. Using a single decoder or multi-decoder network structure, the two are organically integrated into one, thereby achieving speech enhancement with lower model complexity, stronger model interpretability, and better noise reduction performance.

[0032] In some optional embodiments, the step of using the dual-path module to model the preset features on the frequency axis to obtain frequency axis features includes: performing a first matrix dimension transformation on the preset features; inputting the features after the first matrix dimension transformation into the frequency sequence layer for processing; performing a second matrix dimension transformation on the features output by the frequency sequence layer; and performing layer normalization on the features after the second matrix dimension transformation to obtain frequency axis features.

[0033] In a further optional embodiment, the step of using the dual-path module to model the frequency axis features on the time axis to obtain time axis features includes: performing a first matrix dimension transformation on the preset features; inputting the features after the first matrix dimension transformation into the time series layer for processing; performing a second matrix dimension transformation on the features output from the time series layer; and performing layer normalization on the features after the second matrix dimension transformation to obtain time axis features. The skip connection can more effectively transmit information from the encoder module to the decoding module.

[0034] In a further optional embodiment, there is a skip connection between the input of the frequency axis modeling and the output of the frequency axis modeling, and a skip connection between the input of the time axis modeling and the output of the time axis modeling.

[0035] In some optional embodiments, the encoder module includes M two-dimensional convolutional modules, each of which includes a two-dimensional deconvolutional layer, a batch normalization layer, and a PReLU activation layer; each decoder in the decoder module includes M two-dimensional deconvolutional modules, each of which includes a two-dimensional deconvolutional layer, a batch normalization layer, and a PReLU activation layer.

[0036] In a further optional embodiment, each of the two-dimensional convolutional modules and its corresponding two-dimensional deconvolutional module are skip-connected via a convolutional attention module. The convolutional attention module includes a channel attention submodule and a frequency attention submodule. The channel attention submodule processes data along the channel axis, and the frequency attention submodule processes data along the frequency axis. The outputs of both the channel attention submodule and the frequency attention submodule are obtained by element-wise multiplication of matrices. Skip-connecting with convolutional attention modules allows the neural network to selectively transmit information through learning.

[0037] In a further optional embodiment, during training, the speech enhancement system calculates the cost function error between the estimated speech signal and the expected clean speech signal, and uses backpropagation to update the neural network model parameters of the speech enhancement system. During training, the training data is randomly shuffled in each round to iteratively update the model parameters until the cost function error no longer decreases significantly, at which point training stops.

[0038] In a further optional embodiment, the cost function includes adding a time-domain L1 cost function to a multi-scale short-time Fourier cost function.

[0039] It should be noted that the above method steps are not intended to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the reverse order of the steps, and this application does not impose any restrictions on this.

[0040] The following description addresses some problems encountered by the inventors in realizing this invention and provides a specific embodiment of the final solution, so that those skilled in the art can better understand the solution of this application.

[0041] The inventors discovered that the aforementioned defects are mainly caused by the following reasons: Speech enhancement methods based on signal processing suffer from defects due to noise tracking and estimation algorithms. Overestimation of the noise energy spectrum leads to speech distortion, while underestimation of the noise energy results in residual noise. On one hand, neural network denoising methods perform denoising in the short-time Fourier transform domain or some network transform domain, without considering the physical generation model of the speech signal, thus exhibiting poor modeling stability and requiring massive amounts of training data. On the other hand, speech enhancement methods based on vocoder synthesis, while considering the physical generation model of the speech signal, only process the amplitude spectrum of the speech signal, which limits the upper limit of the algorithm's performance.

[0042] To address the shortcomings of high algorithm complexity and poor generalization ability in neural networks, current academic and industrial researchers generally employ more sophisticated neural network structures, more complex cost functions, and larger amounts of training data. However, these methods do not consider the inherent characteristics of the speech signal itself, typically estimating only the time-frequency mask in the time-frequency domain or directly estimating the pure speech signal. While vocoder-based speech enhancement methods, whether traditional or neural network vocoders, consider the characteristics of the speech signal, their algorithmic complexity remains high, making them difficult to apply in engineering practice.

[0043] The speech enhancement method, which draws inspiration from neural network vocoders, combines the advantages of neural network denoising and acoustic feature representation, achieving a speech enhancement method with relatively low algorithm complexity and good denoising performance to a certain extent. However, it typically only processes the amplitude spectrum of the speech signal, and the denoised speech signal can only use the noisy phase or the implicitly estimated phase. This processing method limits the performance ceiling of the denoising algorithm to some extent.

[0044] To address the aforementioned shortcomings, this application integrates the complex spectrum neural network denoising method into the neural homomorphic synthesis speech enhancement method. By using a single-decoder and multi-decoder network structure, the two are organically combined, thereby achieving speech enhancement with lower model complexity, stronger model interpretability, and better denoising performance.

[0045] Please refer to Figure 2This document illustrates a system block diagram of a speech enhancement method integrating neural homomorphic synthesis and time-frequency mask estimation, provided in an embodiment of this application. It integrates neural homomorphic synthesis and time-frequency mask estimation into a single system. The framework primarily includes an encoder module, a dual-channel signal modeling module, three decoder modules, and a signal merging module. Using only the outputs of the excitation decoder and the vocal tract decoder constitutes a cepstral analysis and synthesis-based neural homomorphic synthesis speech enhancement method; using only the output of the complex decoder constitutes a time-frequency mask estimation speech enhancement method. The following section discusses… Figure 2 The various modules will be introduced below:

[0046] The encoder module takes noisy speech signals as input and outputs transformed features. It is used to extract the time-frequency features of the input speech signals and is generally implemented using a convolutional neural network. The specific network structure will be introduced later.

[0047] The dual-path module takes the encoder output as input and outputs the processed features. It is used to model time-series signals and processes them on the time axis and frequency axis respectively, hence the name dual-path module. The specific network structure will be introduced later.

[0048] The system consists of three decoder modules: an excitation decoder, a channel decoder, and a complex spectrum decoder. The input is the output of a dual-channel module, which respectively models the signal excitation, channel, and complex spectrum. The networks for the three decoders are identical; their specific structures will be described later. The outputs of the excitation decoder and the channel decoder are convolved to obtain the minimum-phase speech signal, which is then combined with the noisy phase signal to obtain one branch of the denoised speech signal. The output of the complex spectrum decoder is masked with the noisy spectrum to obtain another branch of the denoised speech signal.

[0049] The merging module takes the denoised signals from the two branches and the original noisy speech signal as input, and obtains the final estimated speech signal after passing through the merging module. The network structure of this module is similar to that of the encoder, and it is also implemented using a convolutional neural network.

[0050] besides, Figure 2 The preprocessing and post-processing modules are omitted. The preprocessing module includes windowing and framing of the signal, as well as Fourier transform to the frequency domain. For example, for speech with a sampling rate of 16kHz, Hamming windowing is used for framing, with a frame length of 400 sampling points (i.e., 25ms) and a frame shift of 100 sampling points (i.e., 6.25ms). Then, a 512-point Fast Fourier Transform is used to obtain the Fourier frequency domain signal. The post-processing module includes overlapping and addition, as well as an inverse Fourier transform process, to finally obtain the time-domain speech signal.

[0051] Continue to refer to Figure 3It shows a specific structure diagram of the neural network used in the embodiments of this application.

[0052] like Figure 3 As shown, the encoder includes M two-dimensional convolutional modules, the dual-channel module includes N dual-channel sequence modules, and the decoder includes M two-dimensional deconvolutional modules. Each two-dimensional convolutional module and each two-dimensional deconvolutional module are connected via a convolutional attention module. The following sections describe each module in detail:

[0053] The two-dimensional convolution module includes a two-dimensional convolutional layer, a batch normalization layer, and a PReLU activation layer.

[0054] The two-dimensional deconvolution module is similar to the two-dimensional convolution module, and includes a two-dimensional deconvolution layer, a batch normalization layer, and a PReLU activation layer.

[0055] The dual-path sequence module (i.e., the dual-path module) first models the frequency axis, including a matrix transformation step, a frequency sequence layer, another matrix transformation, and layer normalization. Then, it models the time axis, following a similar process: a matrix transformation step, a time series layer, another matrix transformation, and layer normalization. The dual-path sequence module is a cascaded structure, with its input being the output of the previous module. Furthermore, there is a jump connection between the input and the output of the frequency axis modeling, and another jump connection between the input and output of the time axis modeling. Both jump connections use matrix element-wise addition. The purpose of these jump connections is to more effectively transmit information, and they are performed in two different dimensions.

[0056] The convolutional attention module includes channel attention and frequency attention. These two sub-modules process the data on the channel axis and frequency axis, respectively, and the output of each sub-module is obtained by multiplying the matrix elements. In this embodiment, all data dimensions of the neural network are B*C*F*T, that is, batch*channel*frequency*time, where the channel axis is the second axis.

[0057] The speech enhancement method proposed in this application has been validated on the VoiceBank+DEMAND dataset and the DNS Challenge dataset, both commonly used in the field. It mainly consists of the following steps:

[0058] 1. Training Data Preparation Phase

[0059] The VoiceBank+DEMAND dataset contains pre-mixed signal and noise data, with a separate test set. Only a small portion (e.g., 1000 voice samples) of the training set needs to be used as a validation set during training. The DNSChallenge dataset provides only clean speech and noise data, along with pre-mixed test data. Therefore, during data preparation, 300 hours of reverberation-free training data and 200 hours of reverberation-containing data should be generated. The mixed signal-to-noise ratio should be a random value ranging from -5dB to 20dB, and 1000 voice samples should be randomly selected as the validation set for training.

[0060] 2. Model Training Phase

[0061] During the model training phase, noisy speech is used as... Figure 2 The input signal passes through an encoder, a dual-path module, a decoder, and a signal combining module to obtain an estimated clean speech. The estimated speech is compared with the expected clean speech to calculate the cost function error. This error is then used to update the neural network parameters using backpropagation. In each round of training, the training dataset is randomly shuffled, and the model parameters are iteratively updated. Training stops when the error of the cost function on the validation set no longer decreases significantly.

[0062] The model training uses a multi-scale short-time Fourier cost function, the formula of which is:

[0063]

[0064] in, x(n) and x(n) represent the estimated speech and expected clean speech output by the neural network, respectively, ||.||1 and |||.|||. F These represent the L1 norm and the F norm, respectively. Additionally, to improve the time-domain signal-to-noise ratio, a time-domain L1 cost function is added to the final cost function:

[0065]

[0066] In the formula, I represents the set of Fourier transforms.

[0067] 3. Speech Enhancement Model Inference Stage

[0068] After the model achieves the expected results on various test metrics (such as PESQ, SNR, etc.) on the test set, the system and model are engineered and deployed to devices or servers for speech enhancement inference. The process follows... Figure 2 As shown, the input is noisy speech, and the output is the denoised speech.

[0069] In order to compensate for the deficiency of neural homomorphic synthesis denoising which only performs amplitude spectrum processing, the embodiments of this application consider the following two solutions:

[0070] 1. By performing phase processing separately through another cascaded module and recovering the phase only from the amplitude spectrum, the experimental results show that this module is difficult to train, the recovered phase is not accurate enough, and the reconstructed speech quality is poor.

[0071] 2. Using time-frequency mask denoising as a preprocessing or postprocessing module for neural homomorphic synthesis denoising, i.e., a cascaded approach, will significantly increase model complexity, and training two-level models will be more difficult.

[0072] The embodiment of this application achieves a more streamlined network structure by sharing an encoder module and an intermediate dual-path module. Each decoder module performs its own function, and finally, the fusion of multi-branch signals is achieved through a merging module.

[0073] This application integrates a complex-domain time-frequency mask estimation method and a neural homomorphic synthesis denoising method into a single whole, rather than simply cascading them. This approach represents a novel attempt in the field of speech enhancement, and similar solutions may emerge, such as fusing speech enhancement methods from other vocoders with the complex-domain time-frequency masking method, thereby addressing the problem that vocoder speech enhancement methods cannot explicitly estimate the phase.

[0074] The following specific experiments and experimental data verify the beneficial effects of the embodiments of this application compared to the prior art.

[0075] Speech enhancement refers to suppressing background noise to improve the perceptual quality and intelligibility of noisy speech. In recent years, speech enhancement algorithms based on deep neural networks (DNNs) have replaced traditional statistical signal processing-based algorithms and become the mainstream in the research field. However, most DNN-based speech enhancement methods typically operate in the frequency domain without using a speech generation model, making these models prone to insufficient noise suppression or excessive speech suppression. To address this deficiency, this application proposes a novel speech enhancement method that integrates neural homomorphic synthesis and complex spectral masking. Specifically, this application uses a neural network architecture with a shared encoder and multiple decoders. For neural homomorphic synthesis, the speech signal is decomposed into excitation and vocal tract parts through homomorphic filtering. Two DNN decoders are used to estimate the target signal separately, and the denoised speech is synthesized using the estimated minimum phase signal and the noisy phase. For complex spectral masking, another DNN decoder is used to estimate the complex mask of the target speech, and the denoised speech is obtained through a masking operation. These two branches estimate the speech signal separately, and the final enhanced speech is obtained by combining the speech from these two branches through a merging module. Experimental results on two popular datasets show that the method achieves state-of-the-art performance on most evaluation metrics, while the model has only 920K parameters.

[0076] The contributions of this application's embodiments can be summarized as follows:

[0077] 1) This application proposes a novel speech enhancement method that combines the advantages of neural homomorphic synthesis and complex spectral denoising.

[0078] 2) This application presents a novel multi-decoder neural network structure to predict the decomposed speech components and parallel complex spectra.

[0079] 3) The embodiments of this application use complex neural network blocks to construct the network of the embodiments of this application, so that the proposed model has only 920,000 parameters.

[0080] Experimental results on two popular benchmark datasets (i.e., VoiceBank+DEMAND

[36] and DNS Challenge) show that the proposed method achieves state-of-the-art (SOTA) performance on most evaluation metrics and is more competitive in terms of the number of model parameters.

[0081] Neural homomorphic synthesis

[0082] A. Source-Filter Model

[0083] Source filtering models are widely used in digital speech processing. They assume that short-time speech signals are generated by modulation of an excitation signal and a linear time-varying system. Figure 4 A general source filtering model for speech generation is illustrated. The dashed box on the left represents a simplified version of the excitation signal e(n). For silent speech, e(n) is the output of a random noise generator u(n). For spoken speech, e(n) is assumed to be the modulation of a quasi-periodic pulse sequence p(n) and a glottal pulse g(n). The linear time-varying system h(n) is a combination of the glottal and radiating systems. For simplicity, the following embodiments refer to it as the vocal tract system. Therefore, a speech segment x(n) is represented by the convolution of e(n) and h(n):

[0084] x(n) = e(n) * h(n) (1)

[0085] like Figure 4 As shown, e(n) is the excitation signal, h(n) is the vocal tract system, and x(n) is the speech signal.

[0086] Figure 5 Homomorphic analysis of speech is shown. Figure 5 (a) shows the amplitude of the short-time Fourier transform (the dotted line represents the estimated value of the vocal tract spectrum). Figure 5 (b) in the figure shows the estimated value of the excitation spectrum.

[0087] According to the source filter model Figure 5It demonstrates homomorphic analysis of spoken segments. Figure 5 (a) shows the DFT (thin line) of the windowed speech signal, i.e., |FFT{x(n)}|, and the estimate of the vocal tract spectrum (double line), i.e., |FFT{h(n)}|. Figure 5 Figure (b) shows the corresponding estimate of the excitation spectrum, i.e., |FFT{e(n)}|. From the figure, it can be clearly seen that the speech segment can be decomposed into two parts: the excitation and the vocal tract. Details of the deconvolution procedure will be given later in the embodiments of this application.

[0088] B. Short-time frequency reciprocation analysis and synthesis of speech

[0089] Given a speech signal x(n), convolving it with an approximate feature system of Discrete Fourier Transform (DFT) and Inverse DFT (IDFT) yields a complex cepstrum:

[0090]

[0091] Then, in the cepstral domain linear filter (i.e., the elevator):

[0092]

[0093] Where nco is the cutoff threshold, used to... Divided into two parts: the cepstral spectrum of the excitation The inverted spectrum of the vocal cords Then, the time-domain representations of these two components can be obtained through the reverse process:

[0094]

[0095] Finally, by applying a circular convolution of e(n) and h(n), the recovered speech segment is obtained:

[0096]

[0097] Equations (2) and (3) refer to short-time cepstral analysis, while (4) and (5) refer to the corresponding synthesis. Mathematically, these two procedures are completely reversible. That is, x(n) in (2) and x(n) in (5) are completely reversible. Almost identical, Besides the alias "inverted spectrum".

[0098] However, it should be noted that the complex logarithm in (2) has a uniqueness problem. Specifically, given the Fourier transform of x(n), The complex logarithm is:

[0099]

[0100] For the deconvolution procedure, the following formula must be ensured:

[0101]

[0102] In (6), The real part (i.e., log|X(e)) jω The condition is satisfied for arg[X1(e)], but not for the imaginary part. That is, arg[X1(e)] jω )X2(e jω )]≠arg[X1(e jω )]+arg[X2(e jω An effective solution is to use arg[X(e)]. j ω

[38] An additional unwarp function is applied. However, this additional operation produces additional phase-related information, which is equivalent to a circular displacement in the time domain (denoted as n). d Therefore, in the synthesis step, n d It should be added to (4) in the spectral domain or (5) in the time domain.

[0103] Similar to the complex cepstral in (2), the real cepstral is defined as:

[0104] c(n)=IFFT{log|FFT{x(n)}|} (8)

[0105] Then, the booster in (3) is applied to obtain the corresponding real-valued version of the stimulus. Harmony Channel Subsequently, the time-domain minimum phase signal representations of these two components are obtained as follows:

[0106]

[0107] Where w c It is the window for reconstructing the minimum phase signal in the cepstral domain. Similar to (5), the minimum phase signal of the speech segment is recovered:

[0108]

[0109] Finally, the original phase of x(n) was used to synthesize the speech segment:

[0110]

[0111] The main difference between real and complex cue frequency analysis / synthesis is that cue frequency only operates on the amplitude spectrum and uses the original phase to synthesize speech.

[0112] Theoretically, both real and complex cepstrum analysis decompose speech segments into excitations and vocal tracts insensitive to background noise. Let n(n) and y(n) represent additive noise and noisy speech, respectively. Then, y(n) can be expressed as:

[0113]

[0114] By applying (2), we can obtain:

[0115]

[0116] in It is the cepstrum of the noise. Then, the booster in (3) can be applied to obtain the corresponding noise version of the excitation and channel.

[0117] Speech enhancement using neural homomorphic synthesis

[0118] On the other hand, neural vocoders can generate high-fidelity speech from noise-free acoustic features (such as Mel-spectrograms, fundamental frequencies, etc.). However, for noisy acoustic features, the perceptual quality and intelligibility of the synthesized speech drop sharply. In contrast, speech enhancement algorithms can generally achieve acceptable speech quality from noisy speech, even at low signal-to-noise ratios. Therefore, integrating speech denoising models and neural vocoders into a single system holds promise for obtaining high-quality speech enhancement algorithms.

[0119] In related technologies, a complex DNN-based denoiser is integrated into a complex cepstral analysis and synthesized phonocoder. Specifically, we use two complex-valued DNNs to estimate the complex spectrum of the target speech excitation and vocal tract, respectively. The two DNNs are then fused with (4) to obtain:

[0120]

[0121] Where F e {} and F h {} represents two complex-valued DNNs. Furthermore, as discussed earlier, the phase-related information n... d This should also be considered. However, estimating phase-related information is difficult. In previous work, it was not possible to explicitly estimate clean n from noisy speech. d However, complex-valued DNNs are used to handle these issues. That is, phase-related information is implicitly estimated by complex-valued DNNs. While this method significantly improves objective evaluation scores, it is prone to introducing some annoying artificial noise into the enhanced speech.

[0122] To overcome this drawback, a real-valued version of the speech enhancement method based on Neural Homomorphic Synthesis (NHS) (i.e., NHS-MagSE) is proposed in related technologies. Integrating the two DNNs into (9) yields:

[0123]

[0124] Where F e {} and F h {} represents a DNN with two real values. It's important to note that the input to the neural network is the amplitude spectrum, corresponding to... and The real part.

[0125] Figure 6 Cepstral analysis and synthesis block diagram for speech enhancement. 1) Apply cepstral analysis to obtain the cepstral representation, 2) Boost to obtain the excitation and vocal tract, 3) Use cepstral inversion, which includes trainable parameters of the DNN, to obtain the time-domain signal, 4) Apply circular convolution, 5) Use the original phase to synthesize the enhanced speech.

[0126] See the detailed block diagram of NHS-MagSE. Figure 6 For simplicity, preprocessing (i.e., framing) and postprocessing (i.e., overlap addition) are not included in the figure. First, cepstral analysis (8) (including DFT, logarithmic amplitude, and IDFT) is applied to the speech segment. Second, the spectrum is separated using the riser of (3) to obtain the excitation and phonation. Third, two cepstral inversions (15) (including DFT, exponential, forward propagation of DNNs, and IDFT) are used to obtain the temporal representations of the two components respectively. Then, circular convolution (10) is used to obtain the estimated speech signal. Finally, the original phase is used to synthesize the enhanced speech (11).

[0127] Compared to NHS-SE, NHS-MagSE effectively avoids the introduction of synthetic noise. Furthermore, NHS-MagSE uses only real-valued neural networks, which is beneficial for model occupancy. Compared to mainstream DNN-based speech enhancement methods, NHS-MagSE's advantage lies in its utilization of a speech production model, which allows it to effectively model the physical properties of the speech signal, thus achieving better modeling capabilities with the same number of parameters. To fully leverage the advantages of NHS-based speech enhancement algorithms, this paper adopts NHSMagSE as a branch of the ensemble method. The next section will introduce the details of the proposed ensemble speech enhancement method.

[0128] III. Comprehensive Approach

[0129] A. Overview

[0130] The problem of monophonic speech enhancement can be formulated as estimating the clean underground speech x(n) given noise observations y(n). Let x∈R B×W×T y∈R B×W×T , n∈R B×W×T Let B represent a batch of clean speech, noisy speech, and noise, respectively, where B is the batch size, W is the framed window size, and T is the number of frames. The Fourier transforms of x and y are represented as X∈C. B×F×T , Y∈C B×F×T , where F is the number of frequency points.

[0131] Continue to refer to Figure 2 The speech enhancement method described in this application includes a shared encoder, a dual-path block, three decoders, and a merging block. Using only the outputs of the excitation decoder and the channel decoder corresponds to the NHS-MagSE method. Using only the outputs of the complex spectrum decoder corresponds to a popular DNN-based method. The method in this application combines the outputs of both methods.

[0132] Continue to refer to Figure 3 The structure of the neural network is shown. The top left is the encoder-decoder structure, which includes CBAMs (bottom left), dual-channel sequence modules (top right), and Conv2D / ConTrans2D blocks (bottom right).

[0133] exist Figure 2 In the overview of the speech enhancement method shown, the noise input Y in It is the connection between the complex spectrum of the noisy speech and the amplitude spectrum of its decomposed components, i.e., Y in =[Y r ,Y i ,Y e m ,Y h m ]∈R B×C×F×T Where C is 4, Y r and Y i These are the real and imaginary parts of Y. e m and Y h m The amplitude spectra are those of the excitation and the vocal tract, see reference (15). The block diagram contains a shared encoder, several dual-path blocks, three decoders, and a merging block. The three decoders have the same network structure, but their network parameters are independent. For the decoder outputs, using only the outputs of the excitation decoder and the vocal tract decoder corresponds to the NHS-MagSE method, while using only the output of the complex spectrum decoder corresponds to the popular DNN-based method. The merging block combines the two outputs to obtain the final estimated speech.

[0134] B. Neural Network Architecture

[0135] This application uses an encoder-decoder architecture with skip connections to construct deep neural networks. Figure 3 The structure of the neural network is illustrated. The top left shows the framework of the encoder-decoder network, which includes CBAM (bottom left), dual-path blocks (top right), and Conv2D / ConTrans2D blocks (bottom right). Details of each component will be described in subsequent embodiments.

[0136] 1) Encoder and Decoder. The encoder consists of M two-dimensional convolutional (Conv2D) blocks that predict the transformation of input features into the latent space. Similarly, the decoder consists of M two-dimensional convolutional (ConTrans2D) blocks that predict the transformation of features from the latent space back into the original space. Figure 3 The lower right corner shows the details of the Conv2D and ConTrans2D blocks, which consist of Conv2D and ConTrans2D layers respectively, followed by a batch normalization layer and PReLu activation.

[0137] Specifically, the encoder has a block size M of 3, and the number of channels in the three convolutional layers are 16, 32, and 64, respectively. For the first two convolutional layers, the kernel size, span, and padding are (5, 2), (2, 1), and (1, 1), respectively. For the last convolutional layer, the kernel size, span, and padding are (1, 1), (1, 1), and (0, 0), respectively. Therefore, the encoder output is D ∈ R. B×C′×F′×T Where C′=64, The block size M of the three decoders is also set to 3, and the number of channels, kernel size, span, and padding of each ConTrans2D layer are set to be the mirror of each Conv2D layer.

[0138] 2) Convolutional Block Attention Module. This shows that adding a Convolutional Block Attention Module (CBAM) between each encoder and decoder block can effectively improve performance, rather than skipping connections directly, with negligible parameter overhead. Figure 3 The lower left corner shows the Bolck diagram of CBAM, which consists of a channel attention block and a frequency (ffeq.) attention block.

[0139] The channel attention block consists of two parallel pooling layers and a shared linear block with two layers. Let the output of the i-th encoder block be Y. (i) ∈R B×Ci×Fi×T Where i ∈ [1, M]. The channel attention gate G of the i-th block. (i) c ∈R B×Ci×1×1 The calculation method is as follows:

[0140]

[0141] Where σ is the sigmoid activation, and W1 and W2 are the weights of the shared linear block. and These are the average pooling and max pooling operations on Y(i), respectively. The channel attention gate is applied to Y. (i) dot product Y c (i) =G c (i) ·Y (i) .

[0142] The frequency attention block, whose input is the output of the previous channel attention block, consists of two cascaded pooling layers and a Conv2D block. Frequency attention gate G f (i) ∈R B×1×Ci×T The calculation method is as follows:

[0143]

[0144] Where σ is the sigmoid activation and fconv is the Conv2D block. and They are respectively for Y c (i) Average pooling and max pooling operations. A frequency attention gate is applied to the dot product of Y. f (i) =G f (i) ·Y (i) .

[0145] 3) Two-path blocks. Two-path (i.e., two-stage) blocks were initially proposed for speech separation and have proven effective in speech enhancement. Two-path blocks first model the frequency dimension using a sequence neural network block, and then model the time dimension using another sequence neural network block. Figure 5 The upper right corner shows a block diagram of the dual-path sequence model (DP SeqModel) block, which mainly consists of a frequency sequence layer (SeqLayer) and a time series layer. Commonly used sequence layers in dual-path blocks include recurrent neural networks (RNNs), transformers, and shapers. The selection of sequence layers will be described in the experimental section of this application.

[0146] like Figure 3 As shown, the input of the two-path block is the output of the encoder, i.e., D∈R B×C′×F′×T For the frequency path, the embodiments of this application first reshape D into D. F ∈R BT×C′×F′ And a sequence layer is used to simulate the relevant information for each frequency cell on the frequency axis. Subsequently, D FIt is reshaped and normalized using layer normalization. Furthermore, connections skipped from the input are used to speed up convergence. Similarly, the input to the time path is first reshaped into D. T ∈R BF′×C′×T Another sequence layer is applied to capture temporal information on the time axis. Normalization and skip connections are also used in the time path. Generally, the dual-path block models the frequency and time axes alternately, rather than in parallel.

[0147] 4) Merge blocks. For example... Figure 2 As shown, the outputs of the excitation decoder and vocal cord decoder produce one denoised speech branch (i.e., the NHS-branch), and the output of the complex spectrum decoder produces another denoised speech branch (i.e., the Complex-branch). Embodiments of this application apply a merging block in the time domain to merge these two denoised speech samples.

[0148] Let D e ∈RB ×1×F×T D h ∈R B×1×F×T , and D c ∈R B×2×F×T These represent the outputs of the excitation decoder, channel decoder, and complex spectrum decoder, respectively. For the NHS branch, this embodiment applies IDVT and circular convolution to obtain the time-domain speech signal. Then, the noise phase is used to synthesize the denoised speech:

[0149]

[0150] For complex decoders, a masking operation (i.e., dot product) is applied to the noisy speech in the frequency domain to obtain an estimate of the denoised speech:

[0151]

[0152] Where [Y] r ,Y i ] is the noise complex spectrum of channel reduction.

[0153] The embodiments of this application use two denoised speech samples and noisy speech in the time domain to form the input features of the merging block. The merged block is constructed using Conv2D blocks from the encoder. Specifically, the block size is 3, and the number of channels for the three convolutional layers are 3, 3, and 1, respectively. For all three convolutional layers, the kernel size, span, and padding are (5, 3), (1, 1), and (2, 1), respectively. Therefore, the output of the merged block is a weight matrix D. m ∈R B××W×T The enhanced speech frame is obtained by merging the outputs of the NHS branch and the Complex branch with the squeezed weight matrix.

[0154]

[0155] Finally, a batch of one-dimensional time-domain speech signals are obtained through overlap and addition.

[0156] loss function

[0157] In the field of speech enhancement, commonly used loss functions are SNR, scale-invariant SNR (SI-SNR), and mean squared error (MSE). However, multi-scale STFT (MS-STFT) loss can achieve better performance for NHS-based speech enhancement methods. MS-STFT loss was originally designed for training DNN-based vocoders and has proven effective for waveform domain speech enhancement. Xi and Xi represent the estimated speech, respectively. The STFT and FFT of the underground clean speech x(n) in the time domain are of size i∈I. The STFT loss of x(n) is defined as the normalized amplitude spectral distance plus the logarithmic amplitude spectral distance:

[0158]

[0159] Among them / / · / / 1 and / / · / / F These represent the L1 and Frobenius specifications, respectively.

[0160] Recent work has demonstrated the advantages of employing cross-domain loss functions. Combining the joint loss function of the frequency and time domains can further improve speech quality. Therefore, the final loss is formed by adding the estimated speech and clean speech L1 loss to the MS-STFT loss:

[0161]

[0162] Where I is the FFT size set detailed in the experimental section.

[0163] IV. Experiment

[0164] A. Dataset

[0165] This application evaluates the proposed method on two publicly available datasets: VoiceBank+DEMAND dataset 1 and DNS Challenge dataset 2.

[0166] 1) VoiceBank+DEMAND: This is a small dataset widely used to measure the performance of speech enhancement algorithms. Clean speech is collected from the VoiceBank corpus. The original training set has two subsets: one subset contains speakers from 28 English accent regions, and the other subset contains speakers from 56 Scottish and American accent regions. In this embodiment, these two subsets are merged, resulting in a total of 34,647 corpora (approximately 27 hours), and 33,647 corpora are randomly selected for training, with the remaining 1,000 corpora used for validation. The test set contains 872 corpora from 2 unseen British speakers. For both training and validation testing, 8 types of real noise and 2 types of artificial noise from the DEMAND corpus are used to generate mixed corpora with signal-to-noise ratios of 0dB, 5dB, 10dB, and 15dB. For the test set, five unseen noise types from DEMAND were used to generate mixed corpora with different signal-to-noise ratio levels of 2.5dB, 7.5dB, 12.5dB, and 17.5dB. This dataset was recorded and mixed at a frequency of 48kHz; in the experiments of this embodiment, all corpora were down-frequencyd to 16kHz.

[0167] 2) DNS Challenge: This is a large-scale dataset comprising over 500 hours of clean speech from 11,350 speakers, over 60,000 noise clips across 150 categories from Audioset and Freesound, and over 100,000 room impulse response (RIR) clips. For training, this embodiment used synthesis tools provided by related technologies to generate 300 hours of reverberant-free noisy-clean pairs and 200 hours of reverberant noisy-clean pairs, with SNR levels randomly ranging from -5dB to 20dB. Therefore, this embodiment synthesized 60,000 noise-clean pairs and randomly selected 1,000 of them as the validation set. The evaluation set included 150 synthesized noise-clean pairs with and without reverberation, with SNR ranges from 0 to 25dB.

[0168] 3) Data Augmentation. This demonstrates that data augmentation can effectively improve the generalization performance of neural network models, especially for small datasets. Therefore, for the VoiceBank+DEMAND dataset, this embodiment employs the following two data augmentation methods: 1) Remix: Given a batch of noise-clean training pairs (y, x), this embodiment first obtains the noise, n = yx, then shuffles the noise n with the batch axis; finally, this embodiment adds the shuffled noise n′ to the clean speech y′ = x + n′ to obtain a new noise mixture. 2) BandMask: Similar to the SpecAug augmentation method, BandMask uses a band-stop filter to randomly remove 20% of the Mel-scale frequency signals.

[0169] B. Experimental Setup

[0170] 1) Configuration. In the experiments of this embodiment, the speech signal was framed as 400 samples (25ms), using a Hamming window with a jump length of 100 samples (6.25ms), and the FFT size was 512. The block size, i.e., the number of frames fed into the neural network, was set to 637 (4s). The cutoff threshold used to separate the inverted frequencies in (3) was set to 29, which is equivalent to a frequency of 16000 / 29≈550Hz. According to the settings, the number of frequency bins F is 257, and the number of frames F is 637. Unless otherwise specified, the batch size B is set to 12. The number of dual-path blocks N depends on the selection of the sequence layer, which will be described in the following subsection. For the MS-STFT loss in (22), this embodiment uses a Hamming window with a length of {1200, 600, 240}, an overlap rate of 75%, and an FFT size of the next square of the window length.

[0171] 2) Training Details. The weights of all layers were initialized using Xavier and optimized by the AdamW optimizer with an initial learning rate of 0.001. The learning rate was decreased with a factor of 0.5, and a patient wait of 5 epochs was performed while monitoring the validation loss. This embodiment also employed an early stopping strategy, with 11 and 7 epochs respectively for the VoiceBank+DEMAND and DNS Challenge datasets. The maximum training duration for these two datasets was set to 300 and 50 epochs respectively. This embodiment implemented the algorithm in Python using the PyTorchLightning framework, and all experiments were conducted at a high-performance computing (HPC) center.

[0172] Evaluation indicators

[0173] The embodiments of this application use the following measures to evaluate the enhanced speech quality.

[0174] -PESQ: In this embodiment of the application, a wideband version of Perceptual Speech Quality Assessment (PESQ) is selected, with a score range of -0.5 to 4.5.

[0175] -SI-SNR: Scale-invariant signal-to-noise ratio, a time-domain measurement widely used for end-to-end speech separation.

[0176] -STOI / eSTOI: Short-term objective intelligibility or its extended version, extended STOI, with a rating range of 0 to 1.0.

[0177] -SegSNR: Segmented signal-to-noise ratio measurement.

[0178] -CSIG: Comprehensive measurement of signal distortion, following the ITU-T P.835 method, with a scoring range of 0 to 5.

[0179] -CBAK: Similar to CSIG, measures noise distortion.

[0180] -COVL: Similar to CBAK, it measures overall quality.

[0181] For all metrics, a higher score indicates better speech quality.

[0182] D. Research on Neural Network Structures

[0183] This application investigates neural network architectures on the VoiceBank+DEMAND dataset, using PESQ, eSTOI, and SI-SNR as evaluation metrics. The number of model parameters and computational complexity for each model are also examined.

[0184] Figure 7 The spectrograms of noisy (top left) and clean (top right) speech, the estimated excitation (middle right) and vocal cords (middle left), and the output diagrams of the NHS (bottom left) and compound (bottom right) branches of the VoiceBand+DEMAND test set (ID: P232258) are shown.

[0185] Figure 8This document presents a study of neural network architectures on the VoiceBand+DEMAND dataset. The English and Chinese translations are as follows: Noisy: noisy speech; #Params: number of parameters; FLOPs: computational cost; NHS-branch: neural homomorphic synthesis branch; Complex-branch: complex spectral mask branch; Integrated: integrated; block: block; transformer: a sequence model; conformer: a transformer enhanced by convolution; GFLOPS: Giga Floating-point Operations Per Second; PESQ: can be used for encoding / decoding or system evaluation, selection, and optimization; SI-SNR: scale invariant signal-to-noise ratio; STOI: Short-Time Objective Intelligibility.

[0186] 1) Impact of Multi-Branch Structure. This application's embodiments first investigate the impact of the multi-branch structure, with results shown in rows 3-5 of Table 1. NHS-branch, Complex-branch, and Integrated represent the outputs of using only the excitation decoder and channel decoder, using only the complex spectrum decoder, and integrating the outputs of NHS-branch and Complex-branch, respectively. For these three models, this application's embodiments use Long Short-Term Memory (LSTM) RNNs as the building blocks for dual-path blocks, with a block size of 2. The results show that the NHS branch and Complex branch achieve complementary results: the NHS branch achieves a higher score in temporal speech quality measurements (i.e., SI-SNR), while the Complex branch achieves a higher score in frequency-domain speech quality measurements (i.e., PESQ and eSTOI). The integrated method achieves higher scores than both the NHS-branch and Complex-branch methods across all evaluation metrics. Figure 7 The spectrograms of noisy and clean speech are illustrated, along with the estimated excitation and vocal cords, and the outputs of the NHS and Complex branches from the VoiceBand+DEMAND test set (id: p232 258). The figures show that the estimated excitation and vocal cord components provide information about the speech signal, particularly the harmonic structure. The outputs of the NHS and Complex branches provide complementary spectra: the NHS branch provides more detailed information about the speech signal, but it tends to introduce distortions similar to salt and pepper noise; the Complex branch provides a smooth speech spectrum, but it tends to oversuppress the speech. By integrating these two outputs, the oversuppressed spectrum can be compensated for by the output of the NHS branch.

[0187] 2) Selection of Dual-Path Blocks. This application's embodiments investigate the selection of dual-path blocks, and the results are shown in the bottom two rows of Table 1. Transformer and Conformer indicate the use of improved Transformer and Conformer as building blocks for dual-path blocks. For Transformer and Conformer, block sizes N are 4 and 2, resulting in model footprints and computational complexities of 780,000 and 2.49 GFlops, and 870,000 and 10.52 GFlops, respectively. The results show that replacing LSTMRNNs with Transformer modules doubles the computational complexity, without improving PESQ and eSTOI scores, only achieving a slight improvement in SI-SNR. The results also show that compared to LSTM RNNs, Conformer does not improve PESQ and eSTOI scores, and SI-SNR only improves by about 0.5 dB, at the cost of an 8-fold increase in computational complexity. Based on the results of this study, this application's embodiments use LSTM RNNs as building blocks for dual-path blocks in the following experiments.

[0188] E. Comparison with other SOTA methods

[0189] Figure 9 The performance comparison with other state-of-the-art methods on the Voicebank+Demand dataset is shown. '-' indicates that results were not provided in the original paper and the pre-trained model is unavailable. 'with reverb': reverberation; 'no reverb': no ​​reverberation; 'ours': the method proposed in the embodiments of this application.

[0190] Figure 10 The results are shown in comparison with other state-of-the-art methods on the DNS Challenge 2020 test set. '-' indicates that no results were given in the original paper and the pre-trained model was not available.

[0191] 1) Comparison Methods. This application's embodiments select several state-of-the-art (SOTA) DNN-based speech enhancement methods for comparison. Based on the input features of the neural network, the comparison methods can be categorized into temporal domain methods (such as DEMUCS and TSTNN) and TF domain methods (such as MetricGAN / MetricGAN+, DCCRN / DCCRN+, GaGNet, SN-Net, FRCRN, CMGAN, DB-AIAT, FullSubNet, and HGCN+ (these abbreviations are names of relevant models in the prior art and will not be elaborated upon here)). Temporal domain methods directly feed the speech waveform back to the neural network, while TF domain methods first use STFT to obtain the TF representation and then feed the TF features back to the neural network. More specifically, TF domain methods can be further subdivided into large and small spectral methods (such as MetricGAN / MetricGAN+, FullSubNet), complex spectral methods (such as DCCRN / DCCRN+, SN-Net, FRCRN, HGCN+), and composite spectral methods (such as GaGNet, CMGAN, DBAIAT). Amplitude-based spectrum methods operate only on the amplitude spectrum and use noisy phase spectra to synthesize enhanced speech. Complex-based spectrum methods operate on both the real and imaginary components of the complex spectrum. Composite spectrum methods use the real, imaginary, and amplitude components of the complex spectrum to construct input features.

[0192] On the other hand, comparison methods can be divided into single-branch and multi-branch (such as SN-Net, CMGAN, and DB-AIAT) methods. Multi-branch methods use two branches of a neural network to process the respective components of the signal in parallel and combine the two outputs to obtain enhanced speech.

[0193] Based on the above classification and analysis, the method proposed in this application uses complex spectrum and amplitude spectrum of excitation and vocal cords as input features, and merges the outputs of the two neural networks to obtain enhanced speech. It can be classified into the category of multi-branch and is an extension of the composite spectrum method.

[0194] 2) Evaluation of the comparison methods. This application embodiment uses PESQ and STOI as evaluation metrics, and comparative experiments were conducted on two datasets. STOI scores are displayed as percentages. Furthermore, to ensure the comparability of the evaluation results, this application embodiment uses SegSNR, CSIG, CBAK, and COVL for the VoiceBank+DEMAND dataset. Additionally, this application embodiment also uses a narrowband version of PESQ (PESQ-NB) and SI-SNR for the DNS Challenge dataset. Audio samples and supplementary materials are available online.

[0195] Figure 9The evaluation results of the compared methods on the VoiceBank+Demand test set are shown, where '-' indicates that the results were not given in the original paper and the authors did not publish the source code and pre-trained model. The best score for each evaluation metric in the table is indicated in bold. It should be noted that, for fair comparison, the scores of CMGAN in the table are the results without using a discriminator (CMGAN uses MetricGAN as an additional discriminator to optimize the PESQ score). In summary, the results show that the TF domain method outperforms the time domain method, the complex spectrum method outperforms the order-of-magnitude spectrum method, the composite spectrum method outperforms the complex spectrum method, and the multi-branch method outperforms the single-branch method. As can be seen from the results, under the same number of model parameters (i.e., 920,000), the evaluation scores of the proposed method in this application embodiment significantly outperform the TSTNN method in most evaluations. In addition, the proposed method also outperforms the current state-of-the-art methods (i.e., CMGAN and DB-AIAT) in CSIG and COVL.

[0196] CSIG and COVL, while the model parameters are less than half.

[0197] Figure 10 The evaluation results of the compared methods on the DNS Challenge 2020 test set are shown. NSNet2 is the baseline method for the DNS Challenge, which uses a direct recursive network structure and log power spectrum as input features. DCCRN

[30] won first place in the 2020 DNS Challenge, while FRCRN and HGCN+ won second and fifth place in the 2022 DNS Challenge, respectively. In summary, Figure 10 The results and Figure 9 Similar trends were observed, with complex spectral methods (such as DCCRN / DCCRN+) outperforming quantitative spectral methods (such as NSNet2), and multi-branch methods (such as SN-Net) outperforming single-branch methods (such as GaGNet). The results demonstrate that the proposed method achieves competitive performance compared to other state-of-the-art (SOTA) methods. Specifically, the method in this embodiment achieved the highest scores on the reverberationless test sets PESQ-NB and STOI, and the highest scores across all evaluation metrics on the reverberation test set. Furthermore, the method in this embodiment has only 920,000 model parameters. In contrast, other SOTA methods have over 3 million model parameters.

[0198] from Figure 9 VoiceBank+Demand test set and Figure 10 Based on the results of the DNS Challenge test set, the embodiments of this application can draw the following conclusions: 1) The proposed novel multi-branch method improves speech quality more effectively than the single-branch method; 2) The proposed method achieves state-of-the-art (SOTA) performance on most evaluation metrics and is more efficient than other SOTA methods in terms of the number of model parameters.

[0199] V. Conclusion

[0200] In this work, embodiments of this application propose a novel speech enhancement method integrating neural harmonic synthesis and spectral masking. The speech enhancement branch based on neural harmonic synthesis uses homomorphic decomposition to obtain the excitation and vocal cord components of the speech signal, estimates the target amplitude spectrum of the separated components using two DNN modules, and synthesizes denoised speech using the minimum phase signal and noise phase. Simultaneously, the spectral masking branch estimates the mask of the target spectrum and obtains the denoised speech through the composite spectrum of the mask. Finally, the enhanced speech is obtained by merging the outputs of the two branches. Studies of the neural network architecture demonstrate the efficiency and effectiveness of the proposed method. Experimental results on popular datasets VoiceBand+DEMAND and DNS Challenge show that the proposed method achieves state-of-the-art performance on most evaluation metrics.

[0201] In other embodiments, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the speech enhancement method in any of the above method embodiments for use in a speech enhancement system. The speech enhancement system includes an encoder module, a dual-channel module, a decoder module, and a merging module. The decoder module includes an excitation decoder, a channel decoder, and a complex spectrum decoder.

[0202] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0203] The encoder module is used to extract preset features from the original noisy speech signal;

[0204] The preset features are modeled on the frequency axis using the dual-path module to obtain frequency axis features, and the preset features are modeled on the time axis using the dual-path module to obtain time axis features.

[0205] The time axis features are processed using the excitation decoder and the channel decoder to obtain the first denoised speech signal of the time axis branch;

[0206] The frequency axis features are processed using the complex spectrum decoder to obtain the second denoised speech signal of the frequency axis branch;

[0207] The first denoised speech signal, the second denoised speech signal, and the original noisy speech signal are merged using the merging module to obtain the final estimated speech signal.

[0208] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the voice enhancement method and the system. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely configured relative to a processor, and these remote memories may be connected to the voice enhancement method via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0209] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described speech enhancement methods.

[0210] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 11 As shown, the device includes one or more processors 1110 and a memory 1120. Figure 11 Taking a processor 1110 as an example, the device for the voice enhancement method and system may further include an input device 1130 and an output device 1140. The processor 1110, memory 1120, input device 1130, and output device 1140 can be connected via a bus or other means. Figure 11 Taking a bus connection as an example, memory 1120 is the aforementioned non-volatile computer-readable storage medium. Processor 1110 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in memory 1120, thereby implementing the voice enhancement method and system described in the above embodiments. Input device 1130 can receive input digital or character information and generate key signal inputs related to user settings and function control of the voice enhancement device. Output device 1140 may include a display device such as a screen.

[0211] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0212] In one implementation, the above-described electronic device is applied to a speech enhancement system, which includes an encoder module, a dual-channel module, a decoder module, and a merging module. The decoder module includes an excitation decoder, a channel decoder, and a complex spectrum decoder. The system further includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0213] The encoder module is used to extract preset features from the original noisy speech signal;

[0214] The preset features are modeled on the frequency axis using the dual-path module to obtain frequency axis features, and the preset features are modeled on the time axis using the dual-path module to obtain time axis features.

[0215] The time axis features are processed using the excitation decoder and the channel decoder to obtain the first denoised speech signal of the time axis branch;

[0216] The frequency axis features are processed using the complex spectrum decoder to obtain the second denoised speech signal of the frequency axis branch;

[0217] The first denoised speech signal, the second denoised speech signal, and the original noisy speech signal are merged using the merging module to obtain the final estimated speech signal.

[0218] The electronic devices described in this application exist in various forms, including but not limited to:

[0219] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones, multimedia phones, feature phones, and low-end phones, etc.

[0220] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, etc.

[0221] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0222] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0223] (5) Other electronic devices with data interaction functions.

[0224] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0225] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech enhancement method for a speech enhancement system, wherein, The speech enhancement system includes an encoder module, a dual-channel module, a decoder module, and a merging module. The decoder module includes an excitation decoder, a vocal tract decoder, and a complex spectrum decoder. The method includes: The encoder module is used to extract preset features from the original noisy speech signal; The preset features are modeled on the frequency axis using the dual-path module to obtain frequency axis features, and the frequency axis features are modeled on the time axis using the dual-path module to obtain time axis features. The time axis features are processed using the excitation decoder and the channel decoder to obtain the first denoised speech signal of the time axis branch; The frequency axis features are processed using the complex spectrum decoder to obtain the second denoised speech signal of the frequency axis branch; The first denoised speech signal, the second denoised speech signal, and the original noisy speech signal are merged using the merging module to obtain the final estimated speech signal; The step of using the dual-path module to model the preset features on the frequency axis to obtain frequency axis features includes: performing a first matrix dimension transformation on the preset features; inputting the features after the first matrix dimension transformation into the frequency sequence layer for processing; performing a second matrix dimension transformation on the features output by the frequency sequence layer; and performing layer normalization on the features after the second matrix dimension transformation to obtain frequency axis features. The process of using the dual-path module to model and process the frequency axis features on the time axis to obtain time axis features includes: performing a first matrix dimension transformation on the frequency axis features; inputting the features after the first matrix dimension transformation into the time series layer for processing; performing a second matrix dimension transformation on the features output from the time series layer; and performing layer normalization on the features after the second matrix dimension transformation to obtain time axis features.

2. The method according to claim 1, wherein, A first jump connection is included between the input and the output of the frequency axis modeling, and a second jump connection is included between the input and the output of the time axis modeling. Both the first jump connection and the second jump connection use matrix element addition.

3. The method according to claim 1, wherein, The encoder module includes M two-dimensional convolutional modules, each of which includes a two-dimensional deconvolutional layer, a batch normalization layer, and a PReLU activation layer; each decoder in the decoder module includes M two-dimensional deconvolutional modules, each of which includes a two-dimensional deconvolutional layer, a batch normalization layer, and a PReLU activation layer.

4. The method according to claim 3, wherein, Each of the two-dimensional convolutional modules and the corresponding two-dimensional deconvolutional modules are connected by a convolutional attention module. The convolutional attention module includes a channel attention submodule and a frequency attention submodule. The channel attention submodule processes the data along the channel axis, and the frequency attention submodule processes the data along the frequency axis. The outputs of the channel attention submodule and the frequency attention submodule are obtained by multiplying matrix elements.

5. The method according to any one of claims 1-4, wherein, During training, the speech enhancement system calculates the cost function error between the estimated speech signal and the expected clean speech signal. The cost function error is then used to update the neural network model parameters of the speech enhancement system using backpropagation. During each round of training, the training data is randomly shuffled to iteratively update the model parameters until the cost function error no longer decreases significantly, at which point training stops.

6. The method according to claim 5, wherein, The cost function includes a time-domain L1 cost function added to the multi-scale short-time Fourier cost function.

7. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 6.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images