Low-complexity speech enhancement method based on parallel GRU-convolutional neural network

Abstract

The invention discloses a low-complexity speech enhancement method based on a parallel GRU-convolutional neural network, and the method comprises the steps: synthesizing pure speech and noise data, and generating a mixed speech data sample with noise; processing through a Mel filter to obtain Mel logarithmic energy spectrum data characteristics of the mixed voice data; inputting a parallel GRU-convolutional neural network for training to obtain an amplitude value mask of the mixed voice data with noise; performing amplitude estimation according to the amplitude value mask and the amplitude spectrum of the mixed voice data to obtain the amplitude value of the enhanced target voice data; and performing voice reconstruction according to the amplitude value and the phase angle of the enhanced target voice data to obtain enhanced voice data. The method has low complexity and certain speech enhancement performance, and can be deployed in most low-cost processing chips in the market.

Description

Technical Field

[0001] This invention relates to the field of speech enhancement technology, and in particular to a low-complexity speech enhancement method based on a parallel GRU-convolutional neural network. Background Technology

[0002] In the current living environment, many voice acquisition devices are interfered with by a large amount of audio noise in the environment when acquiring the desired voice signal, resulting in a reduced voice signal-to-noise ratio and distortion of details. The purpose of voice enhancement is to suppress noise signals in the environment, preserve and enhance the target voice, so as to improve the quality and listening experience of the voice signal.

[0003] In recent years, neural network-based speech enhancement methods have become the mainstream approach in the field due to their powerful feature learning and nonlinear fitting capabilities. They can more accurately capture the feature differences between speech and noise, and achieve better denoising effects in complex noisy environments. However, in pursuit of high performance, existing neural network speech enhancement models are generally designed by stacking network layers, increasing the number of neurons, and introducing complex modules, resulting in a dramatic increase in the number of parameters and computational complexity. This places high demands on computing power, requiring sufficient memory on high-performance devices to run, and thus their deployment is strictly limited by hardware conditions.

[0004] On the other hand, low-cost, low-power mobile voice acquisition devices (such as portable microphone systems, smartwatches, and small IoT voice terminals) are widely used. These devices mostly employ traditional signal processing and physical noise reduction methods, resulting in limited overall noise reduction effectiveness. Their low-performance chips are constrained by cost and power consumption, with limited memory and computing power, making them unable to meet the operational requirements of high-performance neural network models. Although some perform compression processing such as quantization, pruning, and distillation on the neural network to adapt to low-resource devices, the feature extraction capability and noise suppression accuracy of the compressed model are significantly reduced, resulting in a substantial decrease in voice enhancement performance and making it difficult to match practical application needs.

[0005] Therefore, developing a neural network-based speech enhancement method that combines low complexity with decent speech enhancement performance to meet the application requirements of low-cost, low-power mobile devices has become an urgent technical problem to be solved in this field. Summary of the Invention

[0006] The problem to be solved by this invention is to provide a low-complexity speech enhancement method based on parallel GRU-convolutional neural network, which has both low complexity and certain speech enhancement performance, and can be deployed on most low-cost processing chips on the market.

[0007] This invention adopts the following technical solution: a low-complexity speech enhancement method based on parallel GRU-convolutional neural networks, comprising the following steps:

[0008] S1. Using clean speech and noisy data from publicly available speech datasets, noisy mixed speech data samples are generated through synthesis and divided proportionally to obtain training set, validation set and test set;

[0009] S2. Extract the speech amplitude spectrum from the mixed speech data in the training set, and obtain the Mel logarithmic energy spectrum data features of the mixed speech data through Mel filter processing.

[0010] S3. The Mel logarithmic energy spectrum feature data is input into a parallel GRU-convolutional neural network for training to obtain an amplitude value mask for the noisy mixed speech data. The parallel GRU-convolutional neural network includes a 1D convolutional layer at the input end, which is then divided into two parallel branches: a GRU path and a convolutional path. The GRU path contains two GRU layers, and the convolutional path contains two 1D convolutional layers. The outputs of the two paths are added together and fed into a third GRU layer, then processed by a transposed convolutional layer, and finally the amplitude value mask is output through a sigmoid activation function.

[0011] S4. Estimate the amplitude value based on the amplitude value mask and the amplitude spectrum of the mixed speech data to obtain the amplitude value of the enhanced target speech data. At the same time, directly use the phase of the original mixed noisy speech data as the phase of the target enhanced speech data.

[0012] S5. Reconstruct the speech based on the amplitude and phase angle of the enhanced target speech data to obtain the enhanced speech data.

[0013] As a preferred option, in step S1, a random mixed speech signal-to-noise ratio is set, ranging from [-5, 5], with the unit being dB. The noise data and the clean speech data are mixed according to the set signal-to-noise ratio to obtain noisy mixed speech data, and the training set, validation set, and test set are divided according to a ratio of 7:1:2.

[0014] As a preferred embodiment, step S2 includes the following sub-steps:

[0015] S2.1. Preprocess the mixed speech data by first dividing the speech into frames, then windowing each frame, and finally obtaining the short-time Fourier spectrum through short-time Fourier transform. And take the modulus value of the short-time Fourier spectrum. ;

[0016] in, For frame number, Number of frequency points;

[0017] S2.2 Construct a set of Mel filters, the number of filters being... The square of the modulus of the short-time Fourier spectrum is obtained by passing it through a Mel filter to obtain the Mel spectrum spectrum, and the logarithm is taken to obtain the Mel logarithmic energy spectrum characteristics. .

[0018] As a preferred embodiment, in step S3, the training process of the parallel GRU-convolutional neural network includes:

[0019] The obtained Mel log energy spectrum feature data is fed into the first 1D convolutional layer at the input end, and the result is simultaneously input into the GRU path and the convolutional path;

[0020] The GRU path includes a first GRU layer and a second GRU layer. The output of the first GRU layer is fed into the second GRU layer to obtain the output of the GRU path. The convolution path includes a second 1D convolutional layer and a third 1D convolutional layer. The output of the second 1D convolutional layer is fed into the third 1D convolutional layer to obtain the output of the convolution path.

[0021] The outputs of the GRU path and the convolutional path are directly superimposed and fed into the third GRU layer. The result is then sequentially fed into a 1D transposed convolutional layer and a sigmoid layer to output the final magnitude mask. .

[0022] As a preferred embodiment, the parallel GRU-convolutional neural network uses a scale-invariant negative signal-to-noise ratio (SI-NSNR) loss function, conducts model training using the Adam algorithm, and dynamically adjusts the learning rate.

[0023] As a preferred embodiment, in step S4, the amplitude value mask obtained in step S3 is used. and the short-time Fourier spectrum obtained in step S2 Perform equal-position multiplication to obtain the amplitude spectrum of the enhanced speech data. Simultaneously, the phase angle of the mixed speech data to be processed is obtained. , as the enhanced speech phase angle.

[0024] As a preferred option, in step S5, based on the amplitude spectrum of the enhanced speech data... and phase angle Short-time Fourier spectrum of synthesized and enhanced speech Then, perform an inverse short-time Fourier transform (ISTFT) to obtain the enhanced speech data. .

[0025] The present invention also provides: an electronic device, comprising:

[0026] One or more processors;

[0027] A storage device on which one or more programs are stored;

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-described low-complexity speech enhancement methods based on parallel GRU-convolutional neural networks.

[0029] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0030] 1. The speech enhancement method of this invention selects a deep acoustic feature of speech for network training, so that the network only needs to focus on some low-frequency signal features of speech. The structure adopts a parallel method of GRU and CNN layers to build the model. The GRU module not only has powerful speech temporal modeling capabilities, but also meets the real-time requirements of mobile devices. At the same time, a one-dimensional convolutional layer is selected. Its powerful feature extraction and deconstruction capabilities can effectively improve the speech enhancement performance. The one-dimensional convolution can effectively balance the computational complexity and performance brought by convolution, achieving better network performance with a small sacrifice in performance.

[0031] 2. The speech enhancement method of this invention has a small number of parameters and low complexity, with small model weights, which can be deployed on most low-cost processing chips on the market. The performance of the small full neural network model it uses is strongly correlated with the training data, with a high performance ceiling. It can provide better performance for speech front-end processing devices of certain specific devices, such as conference rooms and office meeting systems. Moreover, during deployment, the model code can be directly constructed by exporting the weight array and using languages ​​such as C++, saving some methods of deploying the model through a framework, thus making it more universal. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the steps of the low-complexity speech enhancement method of the present invention.

[0033] Figure 2 This is a flowchart of the low-complexity speech enhancement method of the present invention;

[0034] Figure 3 This is a flowchart of the feature extraction process for the speech enhancement method of the present invention;

[0035] Figure 4 This is a diagram of the GRU-convolutional neural network structure of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0037] In one embodiment of the present invention, a neural network-based speech enhancement method with low complexity and certain speech enhancement performance is provided, the process of which is as follows: Figure 1 and Figure 2 As shown, it includes:

[0038] S1 uses clean speech and noisy data from a publicly available speech dataset to synthesize the required noisy speech samples, and divides them into training, validation, and test sets according to comparison.

[0039] S2, the speech amplitude spectrum is extracted from the speech data in the mixed training set, and then the Mel logarithmic energy spectrum data features of the mixed speech data are obtained by processing with a Mel filter.

[0040] S3, the Mel logarithmic energy spectrum feature data is input into a parallel GRU-convolutional neural network for training to obtain an amplitude value mask for the noisy mixed speech data; the network includes a 1D convolutional layer at the input end, which is then divided into two parallel branches: a GRU path and a convolutional path: the GRU path contains two GRU layers, and the convolutional path contains two 1D convolutional layers; the outputs of the two paths are added together and fed into a third GRU layer, then processed by a transposed convolutional layer, and finally the amplitude value mask is output through a sigmoid activation function.

[0041] S4. Amplitude estimation is performed based on the amplitude mask and the amplitude spectrum of the mixed speech data to obtain the amplitude value of the enhanced target speech data. At the same time, the phase of the original mixed noisy speech data is directly used as the phase of the target enhanced speech data.

[0042] S5. Speech reconstruction is performed based on the amplitude value and phase angle of the enhanced target speech data to obtain enhanced speech data.

[0043] Specifically, in step S1:

[0044] First, set the desired signal-to-noise ratio (SNR) for the mixed speech, in the range of [-5, 5], in dB.

[0045] Then, the noise data and clean speech data are mixed according to the set signal-to-noise ratio to obtain the required noisy mixed speech data, and the training set, validation set, and test set (not used for training) are divided according to a 7:1:2 ratio.

[0046] Specifically, in step S2, the feature extraction process is as follows: Figure 3 As shown, the method is as follows:

[0047] First, the mixed speech data undergoes basic preprocessing. The speech is first divided into frames. In this embodiment, the frame length is set to 32ms and the frame shift is 8ms.

[0048] Then, windowing is applied to each frame, and the processed data is then subjected to short-time Fourier transform to obtain the short-time Fourier spectrum. , For frame number, The number of frequency points, and the modulus value of it. .

[0049] Next, a set of Mel filters was designed, with the number of filters M set to 80. The magnitude was squared, and the Mel spectrum was obtained by passing the filters through the Mel filters. The logarithm of the spectrum was then used to obtain the Mel logarithmic energy spectrum characteristics. .

[0050] In this embodiment, the transfer function of the Mel filter used is: ;

[0051] in: ; ;

[0052] Among them, parameters Indicates the first The frequency domain sampling point index of a Mel filter. For the short-time Fourier points, Sampling frequency, This represents the conversion function from linear frequency to Mel frequency. Represents the linear frequency of the original speech signal. Indicates the index of the Mel filter. for The inverse function of .

[0053] Furthermore, from the filter transfer function, we obtain: ;

[0054] Specifically, in step S3, the training process of the parallel GRU-convolutional neural network includes:

[0055] The obtained Mel log energy features are fed into the first 1D convolutional layer. The results are simultaneously input into the left GRU path and the right convolutional path. The left GRU path contains the first GRU and the second GRU, while the right path contains the second 1D convolutional layer and the third 1D convolutional layer.

[0056] The output of the first GRU layer on the left is fed into the second GRU layer to obtain the output of the left GRU path. The output of the second 1D convolutional layer on the right is fed into the third 1D convolutional layer to obtain the output of the right convolutional path. The outputs of the two paths are directly superimposed and fed into the third GRU layer. The result is fed into the 1D transposed convolutional layer. Finally, the output is fed into the last sigmoid layer to output the final amplitude mask.

[0057] The input and output dimensions of each network layer are shown in Table 1 below, and the overall network structure is as follows. Figure 4 As shown:

[0058] Table 1. Network Layer Input / Output Dimensions

[0059] Specifically, in this embodiment, the loss function of the parallel GRU-convolutional neural network adopts scale-invariant negative signal-to-noise ratio (SI-NSNR), and the model is trained using the Adam algorithm. The model training iterations are 150 times, with an initial learning rate of 0.001. After 25 iterations, the learning rate is adjusted to 0.0008, after 50 iterations, the learning rate is adjusted to 0.0006, and after 100 iterations, the learning rate is adjusted to 0.0002.

[0060] The loss function is expressed as: ;

[0061] in, , , The data represents the enhanced speech signal, and s represents the clean speech signal.

[0062] Specifically, in step S4:

[0063] Based on the amplitude value mask obtained in S3 and the amplitude spectrum in S2 Equivalent multiplication is performed to obtain the amplitude spectrum of the enhanced speech data. , is represented as: ;

[0064] Simultaneously, the phase angle of the mixed speech data to be processed is obtained. The enhanced speech phase angle is represented as: ;

[0065] in, The function represents the argument function of a complex number, used to extract phase information from the complex spectrum of mixed speech.

[0066] Specifically, in step S5:

[0067] First, based on the enhanced amplitude spectrum... and phase angle Short-time Fourier spectrum of synthesized and enhanced speech ;

[0068] Then, an inverse short-time Fourier transform (ISTFT) is performed to obtain the enhanced speech data. , is represented as: ; .

[0069] To verify the speech enhancement effect of the method of this invention, this embodiment evaluates the overall complexity of the algorithm by the number of model parameters and floating-point operations (FLOPs). The number of parameters determines the memory occupied by the model weights, and the number of operations determines the computing power required for the model to run. Two speech quality evaluation metrics, PESQ and STOI, are also used: one to evaluate speech quality and the other to evaluate speech intelligibility.

[0070] Furthermore, the parallel GRU-convolutional neural network model of the present invention is validated on the test set in S1, and compared with an existing real-time speech enhancement model (DTLN) network model. The model complexity comparison is shown in Table 2, and the enhancement effect comparison is shown in Table 3.

[0071] Table 2 Comparison of Network Complexity

[0072] Table 3 Comparison of network performance on the test set

[0073] As shown in Tables 2 and 3, the model implemented in this invention has a significantly lower number of model parameters and floating-point operations compared to the DTLN network model. Based on the performance comparison, it can be seen that the model implemented in this invention significantly improves the PESQ and STOI metrics for the original noisy speech, while slightly lowering both metrics compared to the DTLN, but the difference is not significant. Therefore, it can be concluded that the method of this invention can achieve good speech enhancement results with a lower number of parameters and less computational power.

[0074] In this embodiment of the invention, an electronic device is also provided, including: one or more processors; a storage device storing one or more programs thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the low-complexity speech enhancement method based on parallel GRU-convolutional neural network described in the above embodiments.

[0075] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A low-complexity speech enhancement method based on parallel GRU-convolutional neural network, characterized in that, The method comprises the following steps: S1, using pure speech and noise data in a speech public data set, generating mixed speech data samples with noise by synthesis, and dividing the data samples into a training set, a validation set and a test set according to a proportion; S2, extracting speech amplitude spectrum of the mixed speech data in the training set, and obtaining mel log energy spectrum data features of the mixed speech data by mel filter processing; S3, inputting the mel log energy spectrum feature data into a parallel GRU-convolutional neural network for training, and obtaining an amplitude value mask of the mixed speech data with noise; the parallel GRU-convolutional neural network comprises a 1D convolutional layer at an input end, and is divided into two parallel branches of a GRU path and a convolutional path after the 1D convolutional layer: the GRU path comprises two GRU layers, and the convolutional path comprises two 1D convolutional layers; outputs of the two paths are added and then input into a third GRU, and finally the amplitude value mask is output through a sigmoid activation function after being processed by a transposed convolutional layer; S4, performing amplitude value estimation according to the amplitude value mask and the amplitude spectrum of the mixed speech data, obtaining an amplitude value of enhanced target speech data, and directly using a phase of original mixed speech data with noise as a phase of the target enhanced speech data; S5, performing speech reconstruction according to the amplitude value and the phase angle of the enhanced target speech data, and obtaining enhanced speech data.

2. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 1, characterized in that, In step S1, a random mixed speech signal-to-noise ratio is set, the range is [-5, 5] and the unit is db, the noise data and the pure speech data are mixed according to the set signal-to-noise ratio, the mixed speech data with noise is obtained, and the training set, the validation set and the test set are divided according to a 7:1:2 proportion.

3. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 1, characterized in that, Step S2 comprises the following substeps, S2.1, pre-process the mixed voice data, frame the voice first, then window each frame of data, then get the short-time Fourier spectrum through short-time Fourier transform , and take the modulus value of the short-time Fourier spectrum ; wherein, is the number of frames, is the number of frequency points; S2.2, construct a set of mel filters, the number of filters is , square the modulus value of the short-time Fourier spectrum, pass through the mel filter to obtain the mel spectrum spectrum, take the logarithm to obtain the mel logarithmic energy spectrum feature : ； wherein is the transfer function of a mel filter.

4. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 3, characterized in that, The transfer function of the mel filter is as follows: ； where the parameters represent the frequency-domain sample index of the th Mel filter, which is calculated as follows: ； ； wherein is the number of short-time Fourier points, is the sampling frequency, denotes the linear-to-mel frequency conversion function, denotes the linear frequency of the original speech signal, denotes the index of the mel filter, is the inverse function of .

5. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 3, characterized in that, The training process of the parallel GRU-convolutional neural network comprises: The obtained mel log energy spectrum feature data is input into the first layer 1D convolutional layer at the input end, and the obtained result is input into the GRU path and the convolutional path at the same time; The GRU path comprises a first layer GRU and a second layer GRU, the result output by the first layer GRU is input into the second layer GRU, and the output of the GRU path is obtained; the convolutional path comprises a second layer 1D convolutional layer and a third layer 1D convolutional layer, the result of the second layer 1D convolutional layer is input into the third layer 1D convolutional layer, and the output of the convolutional path is obtained; The outputs of the GRU path and the convolution path are directly superimposed, transmitted into a third GRU, and then the obtained results are sequentially transmitted into a 1-dimensional transpose convolution layer and a sigmoid layer, and finally the amplitude value mask is output .

6. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 5, characterized in that, The loss function of the parallel GRU-convolutional neural network adopts a scale-invariant negative signal-to-noise ratio, which is expressed as: ； wherein , , is the enhanced speech signal data, s is the clean speech signal.

7. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 5, characterized in that, The parallel GRU-convolutional neural network is trained by an Adam algorithm, and the learning rate is dynamically adjusted; The number of iterations of the model training is 150 times, the initial learning rate is 0.001, the learning rate is adjusted to 0.0008 after 25 iterations, the learning rate is adjusted to 0.0006 after 50 iterations, and the learning rate is adjusted to 0.0002 after 100 iterations.

8. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 5, characterized in that, In step S4, the amplitude value mask obtained in step S3 is multiplied with the short-time Fourier spectrum obtained in step S2 to obtain an enhanced speech data amplitude spectrum​​ ； At the same time, the phase angle of the mixed language voice data to be processed is taken As the enhanced voice phase angle, it is expressed as: ； wherein The function represents the argument function of a complex number, used to extract phase information from the complex spectrum of the mixed speech.

9. The low-complexity speech enhancement method based on parallel GRU-convolutional neural network according to claim 8, characterized in that, In step S5, the enhanced speech data amplitude spectrum and phase angle are used to synthesize the short-time Fourier spectrum of the enhanced speech, which is expressed as: ； After inverse short-time Fourier transform, the enhanced speech data is obtained is expressed as: ； wherein The function denotes the inverse short-time Fourier transform.

10. An electronic device, comprising: comprise: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the low-complexity speech enhancement method based on the parallel GRU-convolutional neural network as claimed in any one of claims 1 to 9.

Images