A single-channel speech enhancement method based on waveform spectrum fusion network

Abstract

The present application relates to a single-channel speech enhancement method based on waveform spectrum fusion network, which is used to make up for the problem that the frequency domain target decoupling method lacks the use of waveform representation, so that the neural network cannot learn the complementary advantages between time domain and frequency domain. In order to make full use of the information of the two domains of waveform domain and frequency domain, the present application proposes a lightweight semantic aggregation module to fuse the cross-domain features of the two branches, and generates a hierarchical mask tensor through a causal form of mutual attention mechanism. At the same time, the present application proposes a double-path recurrent neural network structure embedded in the neural network to model the long-range context correlation between waveform domain frames and the long-range context correlation within frequency domain frames. The present application is superior to the current most advanced target decoupling type speech enhancement method and the previous advanced system in terms of speech quality and intelligibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech enhancement, and in particular to a single-channel speech enhancement method based on a waveform spectrum fusion network. BACKGROUND

[0002] Single-channel speech enhancement refers to eliminating or suppressing background noise to obtain higher speech quality and intelligibility under the condition of a single microphone. In recent years, deep learning-based speech enhancement methods have achieved excellent results in this regard, especially in challenging scenarios such as non-stationary noise and low signal-to-noise ratio conditions, deep learning methods are significantly superior to traditional single-channel speech enhancement algorithms. However, the single-channel speech enhancement system cannot effectively eliminate the compensation problem between the amplitude spectrum and the phase spectrum.

[0003] In order to alleviate the compensation problem between the amplitude and the phase, people have proposed a target decoupling strategy to decompose the original optimization target into multiple interrelated sub-targets. For this purpose, researchers have designed two effective network architectures in the time-frequency domain, namely multi-stage neural networks and double-branch neural networks. For the former, the neural network jointly optimizes the output of each stage to gradually improve the quality of the enhanced speech. For the latter, the two paths of the network optimize the corresponding target of each path in parallel and cooperatively recover the final spectrum. At the same time, the time-domain method can directly estimate the signal waveform, avoiding the coupling effect between the amplitude and the phase in the complex domain. However, each of the above methods has its drawbacks. On the one hand, implementing short-time Fourier transform on the speech signal can result in a waste of some information, which makes the target decoupling method in the time-frequency domain unable to fully utilize the information in the original speech signal. On the other hand, since the time-domain method ignores the harmonic information which plays an important role in speech quality, the performance bottleneck of the time-domain method cannot be overcome by simply designing different network topologies or training targets.

[0004] Since both the time-domain and time-frequency domain methods have unique advantages, some methods use an interactive learning strategy and propose cross-domain methods. For example, TFT-Net directly maps the complex spectrum to the corresponding waveform, which not only effectively utilizes the harmonic information in the complex spectrum, but also avoids the problem of ineffective short-time Fourier transform. WMPNet uses a multi-stage learning strategy to build a three-stage network to improve speech enhancement performance. However, these methods do not explore the deep correspondence between different semantic feature maps well, and the final resynthesis scheme also does not show the characteristics of the target decoupling strategy, which largely limits the performance.

[0005] CN202210885817.6 A single-channel speech enhancement method based on a progressive fusion correction network only uses the amplitude spectrum features and complex spectrum features in the time-frequency domain for speech enhancement, while the causality cannot be guaranteed, the computational complexity is large, the model parameter amount is large, and it is difficult to deploy in actual terminal systems. Compared with the above method, the present application is a causal form model that can be deployed in actual terminals. In the case of equivalent speech enhancement effect, the computational complexity is lower, the model parameter amount is smaller, and the practicality is stronger.

[0006] CN202210885819.5 A single-channel speech enhancement method based on an interactive time-frequency attention mechanism only uses complex spectrum features in the time-frequency domain, which cannot effectively solve the compensation problem between amplitude and phase. Compared with the above method, the present application simultaneously uses waveform features and complex spectrum features, more fully utilizes the potential information in the speech signal, and the speech enhancement effect is obviously improved. SUMMARY

[0007] The present application solves the problem of overcoming the shortcomings of the prior art and providing a single-channel speech enhancement method based on a waveform spectrum fusion network. In the case of equivalent speech enhancement performance, it can be deployed in actual terminal applications, the computational complexity and model parameters are significantly reduced, and the practicality is stronger. At the same time, the enhancement effect is superior to that of ordinary frequency domain target decoupling type speech enhancement neural networks, and can better solve the problem of speech noise reduction under stationary or non-stationary noise interference.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] In a first aspect, the present application provides a single-channel speech enhancement method based on a waveform spectrum fusion network, comprising the following contents:

[0010] Step 1: Perform short-time Fourier transform (STFT) on the noisy speech signal to obtain the complex spectrum of each frame of the noisy speech signal spectrum, and extract the waveform of each frame;

[0011] Step 2: The complex spectrum is input into the complex spectrum branch of the double-branch network, and the complex spectrum of the input noisy speech signal is subjected to deep feature extraction to obtain a feature map with deep feature information. By adjusting the feature map, the complex spectrum of the enhanced noisy speech signal is recovered, and the enhanced speech signal is preliminarily recovered.

[0012] The waveform is input into the waveform branch of the double-branch network, and the waveform of the input noisy speech signal is subjected to deep feature extraction to obtain a feature map with deep feature information. By adjusting the feature map, a compensation mask for the complex spectrum is generated.

[0013] A semantic aggregation module is introduced between the waveform branch and the complex spectrum branch, which fuses the cross-domain features of the two branches and generates a hierarchical mask tensor through a causal form of mutual attention mechanism, and utilizes the implicit correlation between the waveform branch and the complex spectrum branch;

[0014] The semantic aggregation module is divided into three stages, namely a feature extraction stage, a feature processing stage and a feature fusion stage; in the feature extraction stage, a double-branch neural network is adopted, which is composed of a parallel complex spectrum branch and a waveform branch; the complex spectrum feature extraction path is composed of a complex convolutional encoder in cascade, and the waveform feature extraction path is composed of a waveform convolutional encoder in cascade. The complex spectrum convolutional encoder in the complex spectrum branch performs deep feature extraction on the complex spectrum branch to obtain a feature map of the complex spectrum convolutional encoder; the waveform convolutional encoder in the waveform branch performs deep feature extraction on the waveform branch to obtain a feature map of the waveform convolutional encoder; the outputs of the complex spectrum convolutional encoder and the waveform convolutional encoder are preprocessed in a causal form in the feature extraction stage, and the complex spectrum feature extraction path and the waveform feature extraction path are connected in parallel; the feature fusion stage fuses the feature maps of the complex spectrum feature extraction path and the waveform feature extraction path to obtain a fusion tensor, which is used as a mask for the features of the complex spectrum branch, and the output of the waveform branch is used as a new feature for transmission;

[0015] After inserting a plurality of semantic aggregation modules between the waveform branch and the complex spectrum branch, the noisy speech signal is gradually enhanced according to the feature map after each semantic aggregation module; finally, the last output feature map of the complex spectrum branch represents the estimated complex ideal ratio mask, and the last output feature map of the last semantic aggregation module represents the compensation mask of the estimated complex ideal ratio mask;

[0016] Step 3: Combine the ideal ratio mask and the compensation mask of the ideal ratio mask of the output of step 1 to reconstruct the spectrum of the enhanced speech signal, and then perform inverse short-time Fourier transform on the spectrum of the enhanced speech signal to obtain the enhanced speech signal.

[0017] Optionally, in step 2, the waveform branch includes a real number convolutional encoder, a real number double-branch recurrent neural network model and a real number convolutional decoder; the real number convolutional encoder performs deep feature extraction on the waveform of the input noisy speech signal to obtain a feature map with deep feature information, which is input into the real number double-branch recurrent neural network model to capture the long-range correlation between frames and within frames in the time domain, and obtain a feature map adjusted by the real number double-branch recurrent neural network model; and then the real number convolutional decoder generates a compensation mask for the complex spectrum to reconstruct the enhanced speech signal;

[0018] The complex spectrum branch comprises a complex convolutional encoder, a complex double-branch recurrent neural network model and a complex convolutional decoder; the complex convolutional encoder performs deep feature extraction on the complex spectrum of the input noisy speech signal to obtain a feature map with deep feature information, inputs the feature map with deep feature information into the complex double-branch recurrent neural network model to capture long-range correlation in the time domain and cross-band correlation in the frequency domain, obtains an adjusted feature map after the complex double-branch recurrent neural network model, and then restores the complex spectrum of the enhanced noisy speech signal through the complex convolutional decoder to preliminarily restore the enhanced speech signal.

[0019] Optionally, the real double-branch recurrent neural network model is composed of a real inter-frame recurrent neural network and a real intra-frame recurrent neural network connected in series, the real inter-frame recurrent neural network is used to capture long-range correlation between inter-frame in the time domain in the real domain, and the complex intra-frame recurrent neural network is used to obtain cross-band correlation in the time domain in the real domain.

[0020] The complex double-branch recurrent neural network model is composed of a complex inter-frame recurrent neural network and a complex intra-frame recurrent neural network connected in series, the complex inter-frame recurrent neural network is used to capture long-range correlation in the time domain in the complex domain, and the complex intra-frame recurrent neural network is used to obtain cross-band correlation in the frequency domain in the complex domain.

[0021] Optionally, the complex inter-frame recurrent neural network model is composed of a complex LSTM layer, a complex fully connected layer and a complex layer normalization layer connected in series.

[0022] Let the input of the complex double-branch recurrent neural network model be U in , then the output of the complex inter-frame recurrent neural network model is:

[0023] U mid = CLN(CFC(CLSTM(U in )))+U in (5)

[0024] In the formula, CLN(·), CFC(·) and CLSTM(·) respectively represent a complex layer normalization layer, a complex fully connected layer and a complex LSTM layer, U mid represents the output of the complex inter-frame recurrent neural network model, and U mid is input into the complex intra-frame recurrent neural network after a transformation of the feature dimension.

[0025] The structure of the complex-valued intra-recursive neural network model is the same as that of the complex-valued inter-recursive neural network model, and the output U of the complex-valued intra-recursive neural network is out :

[0026] U out = CLN(CFC(CLSTM(U mid )))+U mid (6)

[0027] The real-valued double-branch recursive neural network model is a real-valued expression form of the complex-valued double-branch recursive neural network model.

[0028] Optionally, the complex convolutional encoder comprises:

[0029] The complex convolutional encoder for feature extraction on the noisy speech spectrum is stacked by six convolutional blocks, each of which is composed of a complex-valued convolutional layer, a complex-valued layer normalization layer and a complex-valued parametric ReLU activation function.

[0030] The complex-valued convolutional layer is simulated by four real-valued convolutional layers according to the operation rules of complex multiplication, assuming that the complex-valued filter matrix W = W r +jW i , the complex-valued input vector X = X r +jX i , wherein W r and W i are real-valued tensor filter matrices, X r and X i are real-valued input tensors, and the real value part is used to simulate complex operation, and the output of the complex convolution operation is represented as:

[0031] F out = (X r *W r -X i *W i )+j(X r *W i +X i *W r ) (2)

[0032] In the formula, F out is the output of the complex-valued convolutional layer; similarly, there are complex-valued LSTM layers and complex-valued fully connected layers, and the outputs F gru and F linear are respectively:

[0033] F gru = (F rr -F ii )+j(Fri +F ir )

[0034] F rr = LSTM r (X r )F ir = LSTM r (X i )

[0035] F ri = LSTM i (X r )F ii = LSTM i (X i ) (3)

[0036] F linear = (F rr - F ii ) + j (F ri + F ir )

[0037] F rr = Linear r (x r )F ir = Linear r (X i )

[0038] F ri = Linear i (X r )F ii = Linear i (X i ) (4)

[0039] In the formula, LSTM and Linear represent LSTM neural network and full connection neural network respectively, and subscripts r and i represent real part and imaginary part of the corresponding network respectively.

[0040] Optionally, in step 2, the semantic aggregation module is implemented as follows:

[0041] In the feature extraction stage, let the input of the complex spectrum feature extraction path and the waveform feature extraction path be Y and y respectively, Y contains C channels, T time frames and F frequency windows; y contains C channels, T time frames, and N points within the frame, f s (·) and f v (·) are mapping functions represented by the complex spectrum feature extraction path and the waveform feature extraction path respectively; the mapping function f s(·) It consists of a complex two-dimensional convolutional layer, a complex batch normalization layer, and a complex activation function connected sequentially, with the waveform feature extraction path mapping function f. v (·) is composed of a gated convolutional layer in real form, a batch normalization layer in real form, and an activation function in real form, connected sequentially; the implicit amplitude spectrum feature A and the implicit waveform feature a output in the feature extraction stage are represented as:

[0042] A = f s (Y) (7)

[0043] a = f v (y) (8)

[0044] The inputs to the feature processing stage are the amplitude spectrum implicit feature A and the waveform implicit feature a, which are output from the feature extraction stage. The complex spectrum feature processing stage uses a T×T dimensional upper trigonometric function to process the amplitude spectrum implicit feature A in a causal form, resulting in the output feature map Q of the complex spectrum feature processing stage, which is represented as follows:

[0045] Q = AK (9)

[0046] In the formula, K is an upper trigonometric function, specifically the value of K in the p-th row and q-th column. pq Represented as:

[0047]

[0048] The waveform feature processing path reconstructs the dimension of the implicit waveform feature 'a' using an average pooling layer and a broadcast layer. The output of the waveform feature processing stage is:

[0049] W=Broadcast(AvgPool(y)) (11)

[0050] In the formula, AvgPool(·) and Broadcast(·) represent the mean pooling layer and the broadcast layer, respectively, and W represents the output of the waveform feature processing stage;

[0051] The feature fusion stage uses a causal attention mechanism to fuse the output Q of the complex spectrum feature processing path and the output W of the waveform feature processing path. The output of the feature fusion stage is represented as:

[0052] M=Q⊙W (12)

[0053] In the formula, ⊙ represents element-wise multiplication. The output of the feature fusion stage is then combined with the input Y of the complex spectrum feature extraction path to obtain the output of the final complex spectrum branch. The output Y' of the final complex spectrum branch is expressed as:

[0054] Y'=M⊙Y (13)

[0055] Optionally, in step 3, the output ideal scaling mask and the compensation mask of the ideal scaling mask are combined to reconstruct the spectrum of the enhanced speech signal, including:

[0056] Predicted output of a complex ideal scale mask for a given complex spectral branch Compensation mask for ideal scaling mask of waveform branch The final spectrum reconstruction method is as follows:

[0057]

[0058] In the formula, Let X represent the complex spectrum of the enhanced speech signal, and X be the complex spectrum of the speech signal to be enhanced. This reconstruction strategy effectively utilizes both the waveform information and the spectral information of the speech signal simultaneously. It solves the problem of compensation between amplitude and phase.

[0059] In a second aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned single-channel speech enhancement method based on a waveform spectrum fusion network.

[0060] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned single-channel speech enhancement method based on a waveform spectrum fusion network.

[0061] The advantages of this invention compared to the prior art are:

[0062] (1) This invention relates to a single-channel speech enhancement method based on a waveform-spectrum fusion network, designed to overcome the problem that frequency-domain target decoupling methods lack utilization of waveform representation, preventing neural networks from learning the complementary advantages between the time and frequency domains. To fully utilize information from both the waveform and frequency domains, this invention proposes a lightweight semantic aggregation module that fuses cross-domain features from two branches and generates a hierarchical mask tensor through a causal mutual attention mechanism. Simultaneously, this invention proposes embedding a dual-path recurrent neural network structure into the neural network to model long-range contextual correlations between waveform domain frames and within frequency domain frames. This invention outperforms state-of-the-art target decoupling speech enhancement methods and previous advanced systems in terms of speech quality and intelligibility.

[0063] (2) This invention employs both complex-form and real-form dual-branch recurrent neural networks to capture long-term contextual relationships in the speech signal in the time-frequency domain. This not only compensates for the shortcomings of previous real-number neural networks in capturing long-term contextual information in the time domain, but also simulates the interrelationships between different frequency bands through the frequency-domain recurrent neural network, ultimately enhancing the noise reduction capability. Compared to a single-channel speech enhancement method based on a progressive fusion correction network, this method uses a causal model that can be deployed in actual terminals. With comparable speech enhancement effects, this patented method has lower computational complexity, fewer model parameters, and greater practicality. Compared to a single-channel speech enhancement method based on an interactive time-frequency attention mechanism, this method utilizes both waveform features and complex spectrum features, making fuller use of the latent information in the speech signal, resulting in a significant improvement in speech enhancement performance.

[0064] (3) This invention introduces a semantic aggregation module embedded in the waveform branch and the complex spectrum branch, fully utilizing the implicit correlation between each layer of the waveform branch and the complex spectrum branch. The semantic aggregation module integrates the cross-domain features of the waveform branch and the complex spectrum branch, and generates a hierarchical mask tensor through a causal mutual attention mechanism to compensate for the amplitude feature information in the complex spectrum branch. Under the condition of comparable speech enhancement effect, it reduces the computational complexity and the number of model parameters.

[0065] (4) This invention proposes a novel time-spectrum reconstruction method to solve the compensation problem between amplitude and phase spectra in traditional neural networks during training. First, a complex ideal proportional mask is predicted through the complex spectrum branch, and then the phase and amplitude spectra are adjusted simultaneously through the waveform branch. This can effectively alleviate the impact of phase estimation on amplitude estimation and ultimately improve the speech enhancement performance of the model. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings and tables used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings and tables described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is the overall network structure of the present invention;

[0068] Figure 2 This describes the specific structure of the semantic aggregation module in this invention;

[0069] Figure 3 This is the specific structure of the complex dual-branch recurrent neural network in this invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0071] like Figure 1 As shown in the embodiment of the present invention, the single-channel speech enhancement method based on waveform spectrum fusion network includes the following steps:

[0072] Step 1: Perform a Short Time Fourier Transform (STFT) on the noisy speech signal to obtain the complex spectrum of each frame in the noisy speech signal spectrum, and simultaneously extract the waveform of each frame. The complex spectrum is input into the complex spectrum branch of the dual-branch network, and the waveform is input into the waveform branch of the dual-branch network;

[0073] The Short-Time Fourier Transform (STFT) includes: downsampling the noisy speech signal so that the sampling rate of all audio signals is 16kHz; performing a STFT with a frame length of 400, a frame shift of 200, and a Hanning window as the window function; and obtaining the real and imaginary parts of each frame in the spectrum of the noisy speech signal, as shown below:

[0074] Y(t,f)=S(t,f)+N(t,f) (1)

[0075] In the formula,

[0076] Y = Y r +jY i

[0077] S = S r +jS i

[0078] Where Y(t,f) represents the single-channel speech spectrum to be enhanced after short-time Fourier transform, t represents the time dimension, and f represents the frequency dimension; S(t,f) and N(t,f) represent clean speech and background noise, Y and S represent the spectra of Y(t,f) and S(t,f), respectively, and the subscripts r and i represent the real and imaginary parts of the spectrum, respectively. The number of short-time Fourier transform points is 512, and the 257-dimensional spectrum after transformation corresponds to a frequency range from 0 to 8000Hz.

[0079] The waveform branch consists of a real-number convolutional encoder, a real-number dual-branch recurrent neural network model, and a real-number convolutional decoder. The real-number convolutional encoder performs deep feature extraction on the input waveform of the noisy speech signal to obtain a feature map with deep feature information. The feature map is input into the real-number dual-branch recurrent neural network model to capture long-range correlations between frames and within frames in the time domain, resulting in a feature map adjusted by the real-number dual-branch recurrent neural network model. The real-number convolutional decoder then generates a compensation mask for the complex spectrum to reconstruct the enhanced speech signal. The system consists of a real-number inter-frame recurrent neural network and a real-number intra-frame recurrent neural network. The real-number inter-frame recurrent neural network is used to capture long-range correlations between frames in the time domain in the real number domain, while the complex-number intra-frame recurrent neural network is used to obtain cross-band correlations within frames in the time domain in the real number domain.

[0080] The complex spectrum branch consists of a complex convolutional encoder, a complex dual-branch recurrent neural network model, and a complex convolutional decoder. The complex convolutional encoder extracts deep features from the complex spectrum input of the noisy speech signal, obtaining a feature map with deep feature information. This feature map is input into the complex dual-branch recurrent neural network model to capture long-range correlations in the time domain and cross-band correlations in the frequency domain, resulting in a feature map after adjustment by the complex dual-branch recurrent neural network model. The complex convolutional decoder then recovers the complex spectrum of the enhanced noisy speech signal, used for preliminary recovery of the enhanced speech signal. The complex dual-branch recurrent neural network model consists of a complex form of inter-frame recurrent neural network and a complex form of intra-frame recurrent neural network. The complex form of inter-frame recurrent neural network is used to capture long-range correlations in the time domain in the complex domain, and the complex form of intra-frame recurrent neural network is used to obtain cross-band correlations in the frequency domain in the complex domain.

[0081] A causal semantic aggregation module is introduced between the waveform and complex spectrum branches to obtain feature maps of both branches after processing. This is implemented in three stages: feature extraction, feature processing, and feature fusion. In the feature extraction stage, the complex spectrum feature extraction path performs deep feature extraction on the complex spectrum branch, resulting in a feature map of the complex spectrum feature extraction path. Similarly, the waveform feature extraction path performs deep feature extraction on the waveform branch, resulting in a feature map of the waveform feature extraction path. In the feature processing stage, causal preprocessing is applied to the outputs of both the complex spectrum and waveform feature extraction paths. In the feature fusion stage, the feature maps of the amplitude spectrum and waveform feature extraction paths are fused to obtain a fusion tensor, which is used as a mask on the features of the complex spectrum branch. The output of the waveform branch is then passed as a new feature.

[0082] After inserting multiple semantic aggregation modules between the waveform branch and the complex spectrum branch, the model progressively enhances the noisy speech signal based on the feature maps obtained from each semantic aggregation module. Finally, the last output feature map of the complex spectrum branch represents the estimated complex ideal scale mask. The last output feature map of the waveform branch represents the compensation mask of the estimated complex ideal scale mask.

[0083] Complex convolution encoders include:

[0084] As an encoder for extracting features from the spectrum of noisy speech, the encoder consists of six stacked convolutional blocks. Each convolutional block comprises a complex convolutional layer, a complex layer normalization layer, and a complex parameterized ReLU activation function. The complex convolutional layer is simulated by four real convolutional layers according to the rules of complex multiplication. Let the complex filter matrix W = W r +jW i The complex form of the input vector X = X r +jX i Among them, W r and W i It is a real tensor filter matrix, X r and X i If the input is a real number tensor, and the real-valued part is used to simulate complex number operations, then the output of the complex convolution operation is represented as:

[0085] F out =(X r *W r -X i *W i )+j(X r *W i +X i *W r (2)

[0086] In the formula, F out It is the output of a complex-form convolutional layer; similarly, there are also complex-form LSTM layers and complex-form fully connected layers, outputting F. gru With F linear They are respectively:

[0087] F gru =(F rr -F ii )+j(F ri +F ir )

[0088] F rr =LSTM r (X r )F ir =LSTMr (X i )

[0089] F ri =LSTM i (X r )F ii =LSTM i (X i (3)

[0090] F linear =(F rr -F ii )+j(F ri +F ir )

[0091] F rr =Linear r (X r )F ir =Linear r (X i )

[0092] F ri =Linear i (X r )F ii =Linear i (X i (4)

[0093] In the formula, LSTM and Linear represent LSTM neural networks and fully connected neural networks, respectively, and the subscripts r and i represent the real and imaginary parts of the corresponding networks, respectively.

[0094] like Figure 2 As shown, the semantic aggregation module includes:

[0095] Step (11) as follows Figure 3 As shown, the complex dual-branch recurrent neural network model consists of a complex inter-frame recurrent neural network and a complex intra-frame recurrent neural network. The complex inter-frame recurrent neural network is used to capture long-range correlations in the time domain in the complex domain, and the complex intra-frame recurrent neural network is used to obtain cross-band correlations in the frequency domain in the complex domain. The complex inter-frame recurrent neural network model is composed of complex LSTM layers, complex fully connected layers, and complex layer normalization layers. The structure of the complex intra-frame recurrent neural network model is the same as that of the complex inter-frame recurrent neural network model. Let the input of the complex dual-branch recurrent neural network model be U. in The output of the complex form of the inter-frame recurrent neural network model is:

[0096] U mid=CLN(CFC(CLSTM(U in )))+U in (5)

[0097] In the formula, CLN(·), CFC(·), and CLSTM(·) represent complex-form layer normalization layer, complex-form fully connected layer, and complex-form LSTM layer, respectively, and U mid This represents the output of the inter-frame recurrent neural network model in complex form. U mid After the feature dimension transformation, the input into the complex form of the intra-frame recurrent neural network results in the following output:

[0098] U out =CLN(CFC(CLSTM(U mid )))+U mid (6)

[0099] In the formula, U out This represents the output of an intra-frame recurrent neural network in complex form. A real-number two-branch recurrent neural network model is the real-number representation of a complex-number two-branch recurrent neural network model.

[0100] In step (12), during the feature extraction stage, a dual-branch neural network is constructed, consisting of a parallel complex spectrum feature extraction path and a waveform feature extraction path. In order to compensate for the amplitude of the complex spectrum, the complex spectrum feature extraction path extracts the amplitude feature information of the complex spectrum that needs to be compensated, and the waveform feature extraction path extracts the waveform feature information used for compensation.

[0101] Let the inputs to the complex spectrum feature extraction path and the waveform feature extraction path be Y and y, respectively. Y contains C channels, T time frames, and F frequency windows; y contains C channels, T time frames, and N intra-frame points, f s (·) and f v (·) represent the mapping functions for the complex spectrum feature extraction path and the waveform feature extraction path, respectively; the mapping function f for the complex spectrum feature extraction path... s (·) It consists of a complex two-dimensional convolutional layer, a complex batch normalization layer, and a complex activation function connected sequentially, with the waveform feature extraction path mapping function f. v (·) is composed of a gated convolutional layer in real form, a batch normalization layer in real form, and an activation function in real form, connected sequentially. The implicit amplitude spectrum feature A and the implicit waveform feature a output in the feature extraction stage are represented as:

[0102] A = f s (Y) (7)

[0103] a = f v (y) (8)

[0104] Step (13) The feature processing stage preprocesses the outputs of the complex spectrum feature extraction path and the waveform feature extraction path in a causal form. This preprocessing is performed by parallel complex spectrum feature processing paths and waveform feature processing paths, cascaded after the complex spectrum feature extraction path and the waveform feature extraction path, respectively. The inputs to the feature processing stage are the amplitude spectrum implicit feature A and the waveform implicit feature a, both output from the feature extraction stage. The complex spectrum feature processing stage uses a T×T dimensional upper trigonometric function to process the amplitude spectrum implicit feature A in a causal form, resulting in the output feature map Q, which can be represented as:

[0105] Q = AK (9)

[0106] In the formula, K is an upper trigonometric function, specifically the value of K in the p-th row and q-th column. pq It can be represented as:

[0107]

[0108] The waveform feature processing path reconstructs the dimension of the implicit waveform feature 'a' using an average pooling layer and a broadcast layer. The output of the waveform feature processing stage is:

[0109] W=Broadcast(AvgPool(y)) (11)

[0110] In the formula, AvgPool(·) and Broadcast(·) represent the mean pooling layer and the broadcast layer, respectively, and W represents the output of the waveform feature processing stage.

[0111] Step (14) Feature fusion stage uses a causal attention mechanism to fuse the output Q of the complex spectrum feature processing path and the output W of the waveform feature processing path. The output of the feature fusion stage can be expressed as:

[0112] M=Q⊙W (12)

[0113] In the formula, ⊙ represents element-wise multiplication. The output of the feature fusion stage is then combined with the input Y of the complex spectrum feature extraction path to obtain the output of the final complex spectrum branch. The output Y' of the final complex spectrum branch can be expressed as:

[0114] Y'=M⊙Y (13)

[0115] Step 2: Combine the ideal scaling mask and the compensation mask of the ideal scaling mask output from Step 1 to reconstruct the spectrum of the enhanced speech signal. Then, perform a short-time inverse Fourier transform on the spectrum of the enhanced speech signal to obtain the enhanced speech signal.

[0116] Predicted output of a complex ideal scale mask for a given complex spectral branch Compensation mask for ideal scaling mask of waveform branch The final spectrum reconstruction method is as follows:

[0117]

[0118] In the formula, Let X represent the complex spectrum of the enhanced speech signal, and let X be the complex spectrum of the speech signal to be enhanced. This reconstruction strategy effectively utilizes both the waveform information and the spectral information of the speech signal.

[0119] The complex ideal scaling mask cIRM is as follows:

[0120]

[0121] The complex ideal scale mask cIRM is used as a training target to reconstruct the time-frequency map of the speech to be enhanced. It is a defined ideal mask where X is the complex spectrum of the speech to be enhanced and S is the complex spectrum of the clean speech signal.

[0122] The table below shows the experimental results of this invention on the open-source speech enhancement dataset Voicebank+Demand.

[0123]

[0124]

[0125] Metrics: Different speech enhancement methods

[0126] Feature: The feature used

[0127] Waveform: Waveform characteristics

[0128] Mag: Amplitude Spectrum Characteristics

[0129] Phase: Phase spectral characteristics

[0130] RI: Complex Spectral Characteristics

[0131] Par.(M): Number of parameters, in millions.

[0132] PESQ: Perceptual Speech Quality Assessment

[0133] STOI: Short-term objective understandability

[0134] CSIG: Subjective Opinion Score, a measure of signal distortion.

[0135] CBAK: Subjective opinion score, a measure of background noise quality.

[0136] COVL: Subjective Opinion Score, a measure of overall audio quality.

[0137] The higher the values ​​of the five indicators PESQ, STOI, CSIG, CBAK, and COVL, the better the sound quality of the speech.

[0138] This invention is the WSFNet method shown in the table above.

[0139] Meanwhile, compared with a single-channel speech enhancement method based on a progressive fusion correction network, the superior performance of the semantic aggregation module and the combined use of the waveform and time-frequency domains result in a reduction of 2.51 million parameters, a reduction of 3.647 billion multiply-accumulate operations, and a 1.39% improvement in real-time performance. Therefore, this invention maintains comparable performance while significantly reducing computational complexity and the number of parameters compared to a single-channel speech enhancement method based on a progressive fusion correction network.

[0140] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0141] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0142] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be defined by the scope of the claims.

Claims

1. A single-channel speech enhancement method based on waveform spectrum fusion network, characterized in that, Includes the following steps: Step 1: Perform a short-time Fourier transform (STFT) on the noisy speech signal to obtain the complex spectrum of each frame in the spectrum of the noisy speech signal, and extract the waveform of each frame at the same time. Step 2: The complex spectrum is input into the complex spectrum branch of the dual-branch network to extract deep features from the complex spectrum of the input noisy speech signal, resulting in a feature map with deep feature information. By adjusting the feature map, the complex spectrum of the enhanced noisy speech signal is recovered, and the enhanced speech signal is initially recovered. The waveform branch in the waveform input dual-branch network performs deep feature extraction on the waveform of the input noisy speech signal to obtain a feature map of deep feature information, and adjusts the feature map to generate a compensation mask for the complex spectrum. A semantic aggregation module is introduced between the waveform branch and the complex spectrum branch. The semantic aggregation module integrates the cross-domain features of the two branches and generates a hierarchical mask tensor through a causal mutual attention mechanism, utilizing the implicit correlation between the waveform branch and the complex spectrum branch. The semantic aggregation module is divided into three stages: feature extraction, feature processing, and feature fusion. In the feature extraction stage, a dual-branch neural network is employed, consisting of parallel complex spectrum and waveform branches. The complex spectrum feature extraction path is formed by cascading complex convolutional encoders, and the waveform feature extraction path is formed by cascading waveform convolutional encoders. The complex spectrum convolutional encoder in the complex spectrum branch performs deep feature extraction on the complex spectrum branch, obtaining the feature map of the complex spectrum convolutional encoder; the waveform convolutional encoder in the waveform branch performs deep feature extraction on the waveform branch, obtaining the feature map of the waveform convolutional encoder. In the feature extraction stage, the outputs of the complex spectrum convolutional encoder and the waveform convolutional encoder are preprocessed in a causal form, consisting of parallel complex spectrum and waveform feature extraction paths, cascaded after the complex spectrum convolutional encoder and the waveform convolutional encoder, respectively. In the feature fusion stage, the feature maps of the complex spectrum feature extraction path and the waveform feature extraction path are fused to obtain a fusion tensor. The fusion tensor acts as a mask on the features of the complex spectrum branch, and the output of the waveform branch is passed as a new feature. After inserting multiple semantic aggregation modules between the waveform branch and the complex spectrum branch, the noisy speech signal is progressively enhanced based on the feature map after each semantic aggregation module. Finally, the last output feature map of the complex spectrum branch represents the estimated complex ideal scale mask, and the last output feature map of the last semantic aggregation module represents the compensation mask of the estimated complex ideal scale mask. Step 3: Combine the ideal scaling mask and the compensation mask of the ideal scaling mask output from Step 1 to reconstruct the spectrum of the enhanced speech signal. Then, perform a short-time inverse Fourier transform on the spectrum of the enhanced speech signal to obtain the enhanced speech signal.

2. The single-channel speech enhancement method based on waveform spectrum fusion network according to claim 1, characterized in that: In step 2, the waveform branch includes a real-number convolutional encoder, a real-number dual-branch recurrent neural network model, and a real-number convolutional decoder. The real-number convolutional encoder performs deep feature extraction on the waveform of the input noisy speech signal to obtain a feature map with deep feature information. The feature map is input into the real-number dual-branch recurrent neural network model to capture long-range correlations between frames and within frames in the temporal domain, resulting in a feature map adjusted by the real-number dual-branch recurrent neural network model. The signal is then processed by a real-valued convolutional decoder to generate a compensation mask for the complex spectrum in order to reconstruct the enhanced speech signal. The complex spectrum branch includes a complex convolutional encoder, a complex dual-branch recurrent neural network model, and a complex convolutional decoder. The complex convolutional encoder performs deep feature extraction on the complex spectrum of the input noisy speech signal to obtain a feature map with deep feature information. The feature map with deep feature information is input into the complex dual-branch recurrent neural network model to capture long-range correlation in the time domain and cross-band correlation in the frequency domain, resulting in a feature map after adjustment by the complex dual-branch recurrent neural network model. The complex convolutional decoder then recovers the complex spectrum of the enhanced noisy speech signal, thus initially recovering the enhanced speech signal.

3. The single-channel speech enhancement method based on waveform spectrum fusion network according to claim 2, characterized in that: The real-number dual-branch recurrent neural network model consists of a real-number inter-frame recurrent neural network and a real-number intra-frame recurrent neural network connected in series. The real-number inter-frame recurrent neural network is used to capture long-range correlations between time-domain frames in the real-number domain, and the complex-number intra-frame recurrent neural network is used to obtain cross-band correlations within time-domain frames in the real-number domain. The complex dual-branch recurrent neural network model consists of a complex inter-frame recurrent neural network and a complex intra-frame recurrent neural network connected in series. The complex inter-frame recurrent neural network is used to capture long-range correlations in the time domain in the complex domain, and the complex intra-frame recurrent neural network is used to obtain cross-band correlations in the frequency domain in the complex domain.

4. The single-channel speech enhancement method based on waveform spectrum fusion network according to claim 3, characterized in that: The complex-form inter-frame recurrent neural network model is composed of complex-form LSTM layers, complex-form fully connected layers, and complex-form layer normalization layers connected in series. Let the input of the complex two-branch recurrent neural network model be U. in Then the output of the complex form of the inter-frame recurrent neural network model is: U mid =CLN(CFC(CLSTM(U in )))+U in (5) In the formula, CLN(·), CFC(·), and CLSTM(·) represent complex-form layer normalization layer, complex-form fully connected layer, and complex-form LSTM layer, respectively, and U mid U represents the output of the inter-frame recurrent neural network model in complex form. mid After transformation of the feature dimensions, it is then input into the complex form of the intra-frame recurrent neural network; The structure of the complex form of the intra-frame recurrent neural network model is the same as that of the complex form of the inter-frame recurrent neural network model. The output U of the complex form of the intra-frame recurrent neural network is... out for: U out =CLN(CFC(CLSTM(U mid )))+U mid (6) The real-number two-branch recurrent neural network model is the real-number representation of the complex-number two-branch recurrent neural network model.

5. The single-channel speech enhancement method based on waveform spectrum fusion network according to claim 2, characterized in that: The complex convolutional encoder includes: The complex convolutional encoder used for feature extraction of the spectrum of noisy speech consists of six stacked convolutional blocks. Each convolutional block is composed of a complex convolutional layer, a complex layer normalization layer, and a complex parameterized ReLU activation function. The complex-form convolutional layer is simulated by four real-form convolutional layers according to the rules of complex multiplication. Let the complex-form filter matrix W = W r +jW i The input vector in complex form X = X r +jX i Among them, W r and W i It is a real tensor filter matrix, X r and X i If the input is a real number tensor, and the real-valued part is used to simulate complex number operations, then the output of the complex convolution operation is represented as: F out =(X r *W r -X i *W i )+j(X r *W i +X i *W r ) (2) In the formula, F out It is the output of a complex-form convolutional layer; similarly, there are also complex-form LSTM layers and complex-form fully connected layers, outputting F. gru With F linear They are respectively: F gru =(F rr -F ii )+j(F ri +F ir ) F rr =LSTM r (X r ) F ir =LSTM r (X i ) F ri =LSTM i (X r ) F ii =LSTM i (X i ) (3) F linear =(F rr -F ii )+j(F ri +F ir ) F rr =Linear r (X r ) F ir =Linear r (X i ) F ri =Linear i (X r ) F ii =Linear i (X i ) (4) In the formula, LSTM and Linear represent LSTM neural networks and fully connected neural networks, respectively, and the subscripts r and i represent the real and imaginary parts of the corresponding networks, respectively.

6. The single-channel speech enhancement method based on waveform spectrum fusion network according to claim 1, characterized in that: In step 2, the semantic aggregation module is specifically implemented as follows: In the feature extraction stage, let the inputs of the complex spectrum feature extraction path and the waveform feature extraction path be Y and y, respectively. Y contains C channels, T time frames, and F frequency windows; y contains C channels, T time frames, and N points within each frame, f s (·) and f v (·) represent the mapping functions for the complex spectrum feature extraction path and the waveform feature extraction path, respectively; the mapping function f for the complex spectrum feature extraction path... s (·) It consists of a complex two-dimensional convolutional layer, a complex batch normalization layer, and a complex activation function connected sequentially, with the waveform feature extraction path mapping function f. v (·) is composed of a gated convolutional layer in real form, a batch normalization layer in real form, and an activation function in real form, connected sequentially; the implicit amplitude spectrum feature A and the implicit waveform feature a output in the feature extraction stage are represented as: A=f s (Y) (7) a=f v (y) (8) The inputs to the feature processing stage are the amplitude spectrum implicit feature A and the waveform implicit feature a, which are output from the feature extraction stage. The complex spectrum feature processing stage uses a T×T dimensional upper trigonometric function to process the amplitude spectrum implicit feature A in a causal form, resulting in the output feature map Q of the complex spectrum feature processing stage, which is represented as follows: Q = AK (9) In the formula, K is an upper trigonometric function, specifically the value of K in the p-th row and q-th column. pq Represented as: The waveform feature processing path reconstructs the dimension of the implicit waveform feature 'a' using an average pooling layer and a broadcast layer. The output of the waveform feature processing stage is: W=Broadcast(AvgPool(y)) (11) In the formula, AvgPool(·) and Broadcast(·) represent the mean pooling layer and the broadcast layer, respectively, and W represents the output of the waveform feature processing stage; The feature fusion stage uses a causal attention mechanism to fuse the output Q of the complex spectrum feature processing path and the output W of the waveform feature processing path. The output of the feature fusion stage is represented as: M=Q⊙W (12) In the formula, ⊙ represents element-wise multiplication. The output of the feature fusion stage is then combined with the input Y of the complex spectrum feature extraction path to obtain the output of the final complex spectrum branch. The output Y' of the final complex spectrum branch is expressed as: Y'=M⊙Y (13).

7. The single-channel speech enhancement method based on waveform spectrum fusion network according to claim 1, characterized in that: In step 3, the output ideal scaling mask and the compensation mask of the ideal scaling mask are combined to reconstruct the spectrum of the enhanced speech signal, including: Predicted output of a complex ideal scale mask for a given complex spectral branch Compensation mask for ideal scaling mask of waveform branch The final spectrum reconstruction method is as follows: In the formula, Let X represent the complex spectrum of the enhanced speech signal, and let X be the complex spectrum of the speech signal to be enhanced. This reconstruction strategy effectively utilizes both the waveform information and the spectral information of the speech signal, solving the compensation problem between amplitude and phase.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1-7.

Images