Voice noise reduction processing method and device

Abstract

The present disclosure relates to a speech noise reduction processing method and device, the speech noise reduction processing method comprising: obtaining a noisy speech signal; performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category; obtaining a noise intensity estimation value according to the first noise-reduced speech signal and the noisy speech signal; and performing signal mixing on the first noise-reduced speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise-reduced speech signal. The second noise-reduced speech signal obtained by the technical solution of the present disclosure alleviates the problem of hearing sensation decline caused by excessive noise filtering, improves the speech quality and overall audio quality, and improves the overall hearing sensation of the speech signal.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of audio processing, in particular to a speech noise reduction processing method and device. BACKGROUND

[0002] Speech noise reduction refers to a technology of removing unnecessary noise in a sound signal to improve speech quality. In real life, due to the complexity and variability of the environment, such as conference rooms, vehicles, streets and other scenes, speech communication or speech recognition technology is often used, and various noises such as wind noise, vehicle noise, machine noise and human voice noise are often contained in these environments. These noises will affect the quality of the speech signal, not only affecting the listening experience, but also causing the accuracy of speech recognition to decrease.

[0003] To solve this problem, speech noise reduction technology has emerged. However, in existing speech noise reduction technology, while filtering out noise, the speech that needs to be preserved is also affected, resulting in the problem of poor listening experience of the final noise-reduced speech. SUMMARY

[0004] Therefore, the present disclosure provides a speech noise reduction processing method and device to improve the effect of speech noise reduction processing and improve the listening experience of the noise-reduced speech signal.

[0005] The technical solution of the present disclosure is implemented as follows:

[0006] A speech noise reduction processing method comprises the following steps:

[0007] obtaining a noisy speech signal;

[0008] performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category;

[0009] obtaining a noise intensity estimation value according to the first noise-reduced speech signal and the noisy speech signal;

[0010] performing signal mixing on the first noise-reduced speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise-reduced speech signal.

[0011] Further, the deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category comprises:

[0012] inputting the noisy speech signal into a short-time Fourier transform (STFT) module, performing STFT on the noisy speech signal through the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum;

[0013] inputting the noisy speech amplitude spectrum into a GRU network, performing feature extraction on the noisy speech amplitude spectrum through the GRU network to obtain a noisy speech feature;

[0014] inputting the noisy speech feature into a first linear classifier to obtain a first feature vector through the first linear classifier;

[0015] inputting the first feature vector into a normalization module to obtain an amplitude spectrum mask through the normalization module;

[0016] inputting the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module to obtain a masked audio amplitude spectrum by multiplying the noisy speech amplitude spectrum and the amplitude spectrum mask through the multiplication module;

[0017] inputting the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module to obtain the first denoised speech signal by performing ISTFT on the masked audio amplitude spectrum and the noisy speech phase spectrum through the ISTFT module;

[0018] inputting the noisy speech feature into a second linear classifier to obtain a second feature vector through the second linear classifier;

[0019] inputting the second feature vector into an average pooling module to obtain a third feature vector by performing average pooling on the second feature vector through the average pooling module;

[0020] inputting the third feature vector into a noise type classifier to obtain the noise category through the noise type classifier.

[0021] Further, the speech denoising processing method further comprises:

[0022] mixing a clean speech sample signal and a noise sample signal to obtain a noisy speech sample signal;

[0023] inputting the noisy speech sample signal into a denoising classification network model composed of the STFT module, the GRU network, the first linear classifier, the normalization module, the multiplication module, the ISTFT module, the second linear classifier, the average pooling module, and the noise type classifier to obtain a noise classification prediction result and a denoised speech prediction signal;

[0024] training the denoising classification network model according to the difference between the noise classification prediction result and a preset noise classification label, and the difference between the denoised speech prediction signal and the clean speech sample signal, to adjust the parameters in the denoising classification network model.

[0025] Further, the difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the pure speech sample signal are used to train the noise reduction classification network model, so as to adjust the parameters in the noise reduction classification network model, including:

[0026] According to the noise classification prediction result and the preset noise classification label, a cross-entropy loss function is established;

[0027] According to the noise reduction speech prediction signal and the pure speech sample signal, a scale-invariant signal-to-noise ratio loss function is established;

[0028] According to the cross-entropy loss function and the scale-invariant signal-to-noise ratio loss function, a total loss function about the noise reduction classification network model is obtained;

[0029] According to the total loss function, the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the noise reduction classification network model are adjusted through back propagation until the loss value of the total loss function converges to an expected value or reaches a training iteration number.

[0030] Further, the noise intensity estimation value is obtained according to the first noise reduction speech signal and the noisy speech signal, including:

[0031] According to the first noise reduction speech signal and the noisy speech signal, a noise signal estimation value in the noisy speech signal is obtained;

[0032] According to the noise signal estimation value and the first noise reduction speech signal respectively, noise signal power and first noise reduction speech signal power are obtained;

[0033] According to the noise signal power and the first noise reduction speech signal power, the noise intensity estimation value is obtained.

[0034] Further, the noise intensity estimation value is obtained according to the noise signal power and the first noise reduction speech signal power, including:

[0035] The signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power is determined as the noise intensity estimation value.

[0036] Further, the noise intensity estimation value and the noise category are used to mix the first noise reduction speech signal and the noisy speech signal to obtain a second noise reduction speech signal, including:

[0037] determining whether signal mixing of the first noise-reduced speech signal and the noisy speech signal is needed according to the noise category;

[0038] in the case that signal mixing of the first noise-reduced speech signal and the noisy speech signal is not needed, not performing signal mixing of the first noise-reduced speech signal and the noisy speech signal; or

[0039] in the case that signal mixing of the first noise-reduced speech signal and the noisy speech signal is needed, performing the following steps:

[0040] obtaining a noise mixing coefficient according to the noise intensity estimation value;

[0041] mixing the first noise-reduced speech signal and the noisy speech signal according to the noise mixing coefficient to obtain the second noise-reduced speech signal.

[0042] Further, the noise category at least includes stationary noise;

[0043] The determining whether signal mixing of the first noise-reduced speech signal and the noisy speech signal is needed according to the noise category comprises:

[0044] in the case that the noise category is the stationary noise, determining that signal mixing of the first noise-reduced speech signal and the noisy speech signal is needed; or

[0045] in the case that the noise category is not the stationary noise, determining that signal mixing of the first noise-reduced speech signal and the noisy speech signal is not needed.

[0046] Further, the obtaining a noise mixing coefficient according to the noise intensity estimation value comprises:

[0047] The noise mixing coefficient is obtained according to the following formula:

[0048]

[0049] wherein β is the noise intensity estimation value, A is a preset lowest value of the noise mixing coefficient, B is a preset highest value of the noise mixing coefficient, m is a preset lower limit value of a noise intensity screening range, and n is a preset upper limit value of the noise intensity screening range.

[0050] Further, the mixing the first noise-reduced speech signal and the noisy speech signal according to the noise mixing coefficient to obtain the second noise-reduced speech signal comprises:

[0051] The second noise-reduced speech signal is obtained through the following formula:

[0052] x_blend = (1 - a) * x_denoise + a * x_noise

[0053] wherein x_blend is the second denoised speech signal, a is the noise mixing coefficient, x_denoise is the first denoised speech signal, and x_noise is the noisy speech signal.

[0054] A speech denoising processing apparatus, comprising:

[0055] a signal obtaining module configured to perform obtaining a noisy speech signal;

[0056] a denoising classification module configured to perform deep denoising processing and classification on the noisy speech signal to obtain a first denoised speech signal and a noise category;

[0057] a noise intensity obtaining module configured to perform obtaining a noise intensity estimate value according to the first denoised speech signal and the noisy speech signal;

[0058] a denoised speech obtaining module configured to perform signal mixing on the first denoised speech signal and the noisy speech signal according to the noise intensity estimate value and the noise category to obtain a second denoised speech signal.

[0059] As can be seen from the above scheme, the speech denoising processing method and apparatus of this disclosure perform deep denoising processing and classification on noisy speech signals, and adaptively mix the denoised speech signal with the original noisy speech signal according to the noise intensity estimate and noise category. This mitigates the problem of decreased listening experience caused by excessive noise filtering. Although the mixing of the denoised speech signal with the original noisy speech signal enhances the performance of the noise component, it also improves the overall speech quality and audio quality, thus improving the overall listening experience of the speech signal. Furthermore, the speech denoising processing method and apparatus of this disclosure utilize an artificial intelligence denoising classification network model to simultaneously perform denoising and noise type prediction, enhancing the correlation between the denoising structure and the prediction results. This helps improve the accuracy of the noise type prediction results in determining whether signal mixing is necessary. Moreover, the mixing coefficient of the speech denoising processing method and apparatus of this disclosure varies according to the magnitude of the noise intensity estimate. When the noise intensity estimate is low, the second denoised speech signal retains more components of the first denoised speech signal and fewer components of the original noisy speech signal. This is because the original noisy speech signal has a higher noise content when the noise intensity estimate is low, significantly impacting the clean speech portion of the denoising process. This results in a higher degree of clean speech removal in the first denoised speech signal. Therefore, the second denoised speech signal retains fewer components of the first denoised speech signal, compensating for the higher degree of clean speech removal in the first denoised speech signal by including more components of the original noisy speech signal. Although some noise is introduced, the performance of the speech component is improved both from the perspective of clean speech and the overall speech signal perspective because it compensates for the higher degree of clean speech removal. This contributes to improving the denoised speech signal. The second denoised speech signal has a better listening experience than the first denoised speech signal. When the noise intensity estimate is high, the second denoised speech signal retains fewer components of the first denoised speech signal and more components of the original noisy speech signal. This is because the original noisy speech signal has lower noise components when the noise intensity estimate is high, and the denoising process has less impact on the clean speech portion. As a result, the degree of filtering out the clean speech portion in the first denoised speech signal is lower. Therefore, the second denoised speech signal retains more components of the first denoised speech signal. By using fewer components of the original noisy speech signal, the performance of the clean speech portion in the first denoised speech signal and the overall performance of the denoised speech signal can be improved, thus helping to improve the listening experience of the denoised speech signal, making the second denoised speech signal better than the first denoised speech signal. Attached Figure Description

[0060] Figure 1 This is a schematic flowchart illustrating a speech noise reduction processing method according to an illustrative embodiment;

[0061] Figure 2 is a modular flowchart diagram of a speech denoising processing method according to an illustrative embodiment;

[0062] Figure 3 is a step flowchart diagram of deep denoising processing and classification of a noisy speech signal according to an illustrative embodiment;

[0063] Figure 4 is a flow topology diagram of deep denoising processing and classification of a noisy speech signal according to an illustrative embodiment;

[0064] Figure 5 is a logical structure diagram of a GRU unit in a GRU network according to an illustrative embodiment

[0065] Figure 6 is a step flowchart diagram of obtaining a noise intensity estimate according to an illustrative embodiment;

[0066] Figure 7 is a step flowchart diagram of signal mixing of a first denoised speech signal and a noisy speech signal according to an illustrative embodiment;

[0067] Figure 8 is an embodiment diagram of a mapping relationship between a noise mixing coefficient and a noise intensity estimate according to an illustrative embodiment;

[0068] Figure 9 is a step flowchart diagram of training a denoising classification network model according to an illustrative embodiment;

[0069] Figure 10 is a step flowchart diagram of adjusting parameters in a denoising classification network model according to a difference during training according to an illustrative embodiment;

[0070] Figure 11 is a step flowchart diagram of a specific application scenario of a speech denoising processing method according to an illustrative embodiment;

[0071] Figure 12 is a logical structure diagram of a speech denoising processing apparatus according to an illustrative embodiment;

[0072] Figure 13 is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, further detailed descriptions will be made to the present disclosure with reference to the accompanying drawings and embodiments.

[0074] It should be noted that the terms "first", "second", and the like in the description and claims of the present disclosure and the foregoing drawings are used to distinguish between similar objects, and do not necessarily have to describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in other sequences than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] Speech noise reduction can use traditional digital signal processing techniques to remove noise, including filter, spectral subtraction, statistical model, etc. In recent years, with the development of deep learning, most of the deep learning algorithms are used to process data with noise and extract pure speech signals. Traditional digital signal processing techniques have good removal effect on steady-state noise (such as long-time sustained wind noise, air conditioner noise, car running noise, etc.), but have poor filtering effect on transient noise (such as door closing sound, siren sound, lightning sound, etc.). Deep learning can better filter out steady-state noise and transient noise, but in the scene of low signal-to-noise ratio and high noise intensity, deep learning may exist in the case of excessive filtering of noise, so that the audio after noise reduction loses some high-frequency or low-frequency information, and further leads to poor listening experience of the speech after noise reduction.

[0076] Therefore, the embodiments of the present disclosure provide a speech noise reduction processing method and device, which uses a deep learning method to reduce noise of a speech signal, and simultaneously predicts a noise type in a neural network structure, and divides the noise type into four categories of steady-state noise, transient noise, human voice noise and other noise. According to the noise type and noise intensity, the speech signal after noise reduction is further processed to improve the effect of the noise reduction audio, and is mixed with the original speech signal to improve the listening experience of the noise reduction audio. The technical solution of the embodiments of the present disclosure can be applied to terminal devices with speech pickup such as monitoring cameras, TWS (True Wireless Stereo, true wireless stereo) earphones, etc.

[0077] Figure 1 is a flowchart of a speech noise reduction processing method according to an illustrative embodiment, Figure 2 is a modular flowchart of a speech noise reduction processing method according to an illustrative embodiment, as Figure 1 and shown in Figure 2 , the speech noise reduction processing method mainly includes the following steps 101 to 104.

[0078] Step 101, obtaining a noisy speech signal.

[0079] In an illustrative embodiment, the noisy speech signal is a speech signal containing background noise, such as a speaking sound collected by an audio collection device. Since the environment in which a person is located is mostly not a quiet environment, the audio collection device will also collect background noise in the environment at the same time as collecting the speaking sound, such as wind, rain, thunder, vehicle driving sound in an outdoor environment, background human speech, walking sound, and movement sound of furniture and other objects in a noisy indoor environment, and the like. When making a voice call or during a video conference, these background noises will be collected by the audio collection device together with the speech signal, thereby obtaining a noisy speech signal.

[0080] Step 102, performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category.

[0081] In an illustrative embodiment, step 102 is implemented using STFT (Short Time Fourier Transform), neural network inference, and ISTFT (Inverse Short Time Fourier Transform). Figure 3 is a step diagram for performing deep noise reduction processing and classification on a noisy speech signal according to an illustrative embodiment, Figure 4 is a flow topology diagram for performing deep noise reduction processing and classification on a noisy speech signal according to an illustrative embodiment, as shown in Figure 3 、 Figure 4 Step 102 mainly includes the following steps 301 to 309.

[0082] Step 301, inputting the noisy speech signal into an STFT module, performing STFT on the noisy speech signal by the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum.

[0083] In an illustrative embodiment, the sampling rate of the noisy speech signal is 16000 Hz, and the dimension of the noisy speech signal is dim=(32, 1, 64000), where 32 is the batch_size size, 1 is the number of channels, and 64000 is the number of speech sampling points (with a sampling rate of 16000 Hz and a time length of 4 seconds as a reference). batch_size represents the size of the batch processing data.

[0084] In an illustrative embodiment, the parameters of STFT are, for example, nfft=320, hop_length=160, and win_length=320, where nfft is the number of Fourier points, hop_length is the step length, and win_length is the window length.

[0085] wherein STFT is a method of analyzing time-domain signals into time-frequency diagrams, and STFT achieves frequency spectrum analysis by dividing a signal into multiple time windows and performing a fast Fourier transform on each window. In STFT, the number of points refers to the number of discrete sampling points used on each window, the step length indicates the time interval for analysis on the input signal, and the window length refers to the length of the time window used when analyzing the signal.

[0086] In an illustrative embodiment, the dimensions of the noisy speech amplitude spectrum and the noisy speech phase spectrum are dim = (32, 161, 400), wherein 32 is the batch size, 161 is the number of channels, and 400 is the time length. Since the step length is set to 160 points, i.e., 10 ms per frame, the time length of a 4 s speech signal is 400. The batch size determines the number of data included in each batch of data input to the GRU network. When training the neural network, the batch size can be adjusted according to the size of the neural network model and the GPU (Graphic Process Unit) memory used. For the Fourier transform, since the Fourier transform results in conjugate complex numbers, the amplitude spectrum is symmetric about the left and right sides, and therefore the number of effective feature points is nfft / 2+1. Since the number of Fourier points is nfft = 320, the number of effective feature points is 320 / 2+1 = 161, i.e., the number of channels.

[0087] In step 302, the noisy speech amplitude spectrum is input to the GRU network, and the noisy speech amplitude spectrum is feature extracted by the GRU network to obtain a noisy speech feature.

[0088] wherein GRU is Gated Recurrent Unit, and the GRU network is a variant of a recurrent neural network (RNN, Recurrent Neural Network) that can be used to process sequence data and model time-dependent relationships.

[0089] Figure 5 is a schematic diagram of the logical structure of a GRU unit in a GRU network according to an illustrative embodiment, as shown in Figure 5 The GRU unit takes the update gate and the reset gate as the core modules, and the input variable x t is concatenated with the state memory variable h t-1 at the previous time point, and the concatenated matrix is input to the update gate after sigmoid nonlinear transformation, to determine the degree to which the state variable at the previous time point is brought into the current state. The reset gate controls the amount of information at the previous time point written into the candidate set, and the h t times the z t-1 stores the information at the previous time point, and the h t times the z record the information of the current time, add the information of the last time and the information of the current time as the output h of the current time t .

[0090] The mathematical description formula of the GRU unit is as follows:

[0091]

[0092] where x t is the input vector, h t-1 is the state memory variable of the last time, h t is the state memory variable of the current time, r t is the state of the update gate, z t is the state of the reset gate, is the state of the current candidate set, y t is the output vector of the current time, W r is the weight parameter of the update gate, W z is the weight parameter of the reset gate, is the weight parameter of the candidate set, W o is y t is the weight parameter of the connection matrix composed of x t and h t-1 , [] represents vector connection, · represents matrix point multiplication, × represents matrix multiplication, σ represents sigmoid activation function, and φ represents tanh activation function. The mathematical description of σ and φ is as follows:

[0093]

[0094]

[0095] In the illustrative embodiment, the GRU network adopts a 3-layer GRU network structure, and the related parameters of the 3-layer GRU network are, for example, input_size = 322, hidden_size = 322, num_layers = 3, and dropout = 0.2. Among them, input_size represents the dimension size of the input feature vector at each time; hidden_size represents the dimension size of the hidden state in the GRU unit, and the hidden state at each time in the GRU network is obtained by a series of operations on the input feature vector at the current time and the hidden state at the previous time, hidden_size determines the dimension size of the hidden state, and can also be understood as the dimension size of the output feature vector of the GRU network at each time; num_layers represents the number of GRU layers in the GRU network, that is, the number of stacked layers of the GRU unit; dropout is a regularization parameter, and regularization is used to prevent overfitting of the GRU network. In the GRU network, dropout can be applied to the positions of the input feature vector, the hidden state, and the output feature vector. According to the value of dropout, some input elements are randomly set to 0, and these elements are discarded with a certain probability, so as to reduce the dependence between neurons, reduce the complexity of the model, and improve the generalization ability of the model. By using dropout in the training process, the model can be forced to learn more robust and robust feature representations, and the overfitting problem can be reduced.

[0096] In step 303, the noisy speech feature is input into the first linear classifier, and a first feature vector is obtained through the first linear classifier.

[0097] In the illustrative embodiment, the first linear classifier can be a convolution with kernel_size = 1. The parameters of the first linear classifier are: kernel_size = 1, input_size = 322, and out_size = 161. Among them, kernel_size refers to the size of the convolution kernel, input_size represents the dimension size of the input feature vector at each time, which is consistent with the input_size of the GRU network, and out_size is the size of the output feature. The size of the output feature of the first linear classifier is set to be equal to the number of channels of the noisy speech magnitude spectrum (x_noisy_mag), so that the number of channels of the subsequent obtained magnitude spectrum mask is equal to the number of channels of the noisy speech magnitude spectrum, so that the noisy speech magnitude spectrum and the magnitude spectrum mask can be matched, and then the multiplication of the noisy speech magnitude spectrum and the magnitude spectrum mask can be realized.

[0098] The first feature vector is the feature extracted by the front GRU network and dimensionally transformed by the first linear classifier. The obtained vector has the same dimension as the amplitude spectrum. Only when the dimension of the first feature vector is the same as that of the amplitude spectrum, the amplitude spectrum mask obtained from the first feature vector can be multiplied with the amplitude spectrum. In addition, the first feature vector is also the feature further extracted by the first linear classifier from the noisy speech feature. Based on this, another purpose of the first linear classifier is also to further extract features.

[0099] Step 304: input the first feature vector into a normalization module to obtain an amplitude spectrum mask through the normalization module.

[0100] In the illustrative embodiment, the normalization module uses a sigmoid function to perform normalization to obtain the amplitude spectrum mask. In the illustrative embodiment, the dimension of the amplitude spectrum mask is dim=(32, 161, 400), which is equal to the dimension of the noisy speech amplitude spectrum.

[0101] Step 305: input the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module to multiply the noisy speech amplitude spectrum and the amplitude spectrum mask to obtain a masked audio amplitude spectrum.

[0102] Through the multiplication in step 305, the amplitude in the corresponding position of the noisy speech amplitude spectrum and the amplitude spectrum mask is changed, so as to achieve the purpose of weakening the amplitude of the noise part.

[0103] Step 306: input the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module to perform ISTFT on the input masked audio amplitude spectrum and the noisy speech phase spectrum through the ISTFT module to obtain a first denoised speech signal.

[0104] ISTFT is the inverse operation of STFT. The parameters used by ISTFT are the same as those used by STFT in step 301 described above, for example: nfft=320, hop_length=160, win_length=320. Thus, the dimension of the obtained first denoised speech signal is the same as that of the noisy speech signal, for example: dim=(32, 1, 64000).

[0105] Step 307: input the noisy speech feature into a second linear classifier to obtain a second feature vector through the second linear classifier.

[0106] In the illustrative embodiment, the second linear classifier can be a convolution with kernel_size = 1. The parameters of the second linear classifier are: kernel_size = 1, input_size = 322, and out_size = 128. Regarding the out_size of the first linear classifier and the out_size of the second linear classifier, the out_size of the first linear classifier is 161 because it must be the same as the dimension of the amplitude spectrum, while the out_size of the second linear classifier is set to 128 because the subsequent noise type classifier has 4 categories (see the subsequent description), and the category is not many, so the dimension of the feature vector in front does not need to be large. The out_size of the second linear classifier is set to 128 because it is an integer multiple of 2, which is easy to process.

[0107] Step 308: input the second feature vector into an average pooling (avg_pooling) module, and obtain a third feature vector by performing average pooling on the second feature vector through the average pooling module.

[0108] In the embodiment of the present disclosure, the second feature vector is reduced from the dimension [32, 128, 400] to [32, 128, 1] through average pooling, that is, the time dimension is changed from 400 to 1.

[0109] Step 309: input the third feature vector into a noise type classifier, and obtain a noise category through the noise type classifier.

[0110] In the illustrative embodiment, the noise type classifier can be a convolution with kernel_size = 1. The parameters of the noise type classifier are: kernel_size = 1, input_size = 128, and out_size = 4. In the illustrative embodiment, the dimension of the noise category is [32, 4], where 32 is the batch size, and 4 represents that the noise is classified into 4 categories. According to the pre-set categories and through training, the required classification result can be obtained, for example, the 4 categories are: stationary noise, transient noise, human voice noise, and other noise. Among them, the stationary noise includes wind noise, air conditioner noise, automobile tire noise, etc., the transient noise includes door closing sound, train whistle sound, thunder and lightning, etc., the human voice noise includes background human voice in a coffee shop, a shopping mall, footstep sound, etc., and the other noise is noise that cannot be classified into stationary noise, transient noise, and human voice noise.

[0111] Step 103: obtain a noise intensity estimation value according to the first denoised speech signal and the noisy speech signal.

[0112] Figure 6 is a flowchart illustrating a step of obtaining a noise intensity estimate according to an exemplary embodiment, as shown in Figure 6 In an exemplary embodiment, step 103 comprises steps 601-603.

[0113] Step 601: obtaining a noise signal estimate in the noisy speech signal according to the first de-noised speech signal and the noisy speech signal.

[0114] In an exemplary embodiment, the noise signal estimate in the noisy speech signal is obtained by subtracting the first de-noised speech signal from the noisy speech signal, since the noise signal is a part of the noisy speech signal, the result obtained by subtracting the first de-noised speech signal from the noisy speech signal is not necessarily the accurate noise signal, but an estimate, so it is referred to as a noise signal estimate. Specifically, the noise signal estimate in the noisy speech signal is obtained by:

[0115] noise2 = x_noise - x_denoise

[0116] where x_noise is the noisy speech signal, x_denoise is the first de-noised speech signal, and noise2 is the noise signal estimate in the noisy speech signal.

[0117] Step 602: obtaining a noise signal power and a first de-noised speech signal power according to the noise signal estimate and the first de-noised speech signal, respectively.

[0118] In an exemplary embodiment, the noise signal power and the first de-noised speech signal power are obtained by:

[0119]

[0120]

[0121] where i is a signal sampling point, N is the total number of sampling points, noise_spe2 is the noise signal power, noise2[i] is the signal amplitude of the i-th signal sampling point of the noise signal estimate in the noisy speech signal, clean_spe2 is the first de-noised speech signal power, and x_denoise[i] is the signal amplitude of the i-th signal sampling point of the first de-noised speech signal.

[0122] Step 603: obtaining a noise intensity estimate according to the noise signal power and the first de-noised speech signal power.

[0123] In an exemplary embodiment, step 603 can specifically comprise: determining the signal-to-noise ratio between the noise signal power and the first de-noised speech signal power as the noise intensity estimate.

[0124] In the illustrative embodiment, the signal-to-noise ratio is obtained by the following formula, i.e. the noise intensity estimate:

[0125]

[0126] wherein snr is the signal-to-noise ratio.

[0127] Step 104, according to the noise intensity estimate and the noise category, signal mixing is performed on the first noise-reduced speech signal and the noisy speech signal to obtain a second noise-reduced speech signal.

[0128] For transient noise such as door closing sound, siren sound, thunder and lightning sound, etc., since the noise duration is very short, the impact on speech is small, and the negative impact of short transient noise on the speech auditory perception of the listener is also small. At the same time, compared with the noisy speech signal containing long-lasting steady-state noise, the speech auditory perception of the first noise-reduced speech signal obtained after the noisy speech signal containing transient noise is noise-reduced is better, in which case, if the first noise-reduced speech signal obtained after the noisy speech signal containing transient noise is noise-reduced is again signal-mixed with the noisy speech signal, the transient noise in the noisy speech signal will not enhance the intelligibility or auditory perception of the speech part, but the transient noise will make the auditory perception of the mixed speech signal worse.

[0129] On the contrary, for steady-state noise such as long-lasting wind noise, air conditioner noise, automobile engine noise, tire noise, etc., since the noise duration is very long, even reaching or exceeding the speech duration, the impact is large, and the negative impact of the continuous steady-state noise on the speech auditory perception of the listener is large. At the same time, compared with the noisy speech signal containing transient noise, the speech auditory perception of the first noise-reduced speech signal obtained after the noisy speech signal containing steady-state noise is noise-reduced will be poor due to the excessive filtering of noise, in which case, if the first noise-reduced speech signal obtained after the noisy speech signal containing steady-state noise is noise-reduced is again signal-mixed with the noisy speech signal to a certain extent, although certain amplification will occur in the noise part, the speech part will also be amplified at the same time, due to the superposition of the speech in the noisy speech signal and the first noise-reduced speech signal, the auditory perception performance of the speech part in the mixed signal will be further enhanced, thus, as a whole, the first noise-reduced speech signal obtained after the noisy speech signal containing steady-state noise is noise-reduced is again signal-mixed with the noisy speech signal to a certain extent, a second noise-reduced speech signal can be obtained.

[0130] The perceptual performance of the voice signal mainly focuses on three aspects: first, the voice quality in the signal; second, the background noise quality in the signal; and third, the overall audio quality. If the voice quality in the signal is high, the perceptual performance will be improved, but excessive filtering of noise by the noise reduction processing will reduce the voice quality and the overall audio quality, although the noise reduction processing removes the background noise to improve the background noise quality, but the decrease of the voice quality and the overall audio quality will lead to the decrease of the perceptual performance.

[0131] In the case that the noise in the noisy voice signal is steady-state noise, the signal mixing of the first noise-reduced voice signal and the noisy voice signal will reduce the noise reduction quality of the background noise in the signal, but the voice quality in the signal will be enhanced, and the overall audio quality will also be improved, so the perceptual performance of the voice signal can still be improved as a whole.

[0132] Figure 7 Fig. 1 is a schematic diagram of a step flow of signal mixing of a first noise-reduced voice signal and a noisy voice signal according to an illustrative embodiment. As shown in Fig. 1, in the illustrative embodiment, step 104 includes steps 701-7032. Figure 7

[0133] Step 701: determining whether the signal mixing of the first noise-reduced voice signal and the noisy voice signal is needed according to the noise category.

[0134] In the illustrative embodiment, the noise category at least includes steady-state noise, and in the illustrative embodiment, the noise category can also include transient noise and other categories, and step 701 specifically includes the following two cases:

[0135] In the case that the noise category is steady-state noise, it is determined that the signal mixing of the first noise-reduced voice signal and the noisy voice signal is needed; or

[0136] In the case that the noise category is not steady-state noise, it is determined that the signal mixing of the first noise-reduced voice signal and the noisy voice signal is not needed.

[0137] Step 702: in the case that the signal mixing of the first noise-reduced voice signal and the noisy voice signal is not needed, the signal mixing of the first noise-reduced voice signal and the noisy voice signal is not performed.

[0138] Step 703: in the case that the signal mixing of the first noise-reduced voice signal and the noisy voice signal is needed, the following steps are performed:

[0139] Step 7031: obtaining a noise mixing coefficient according to the noise intensity estimation value;

[0140] ​Step 7032, mixing the first de-noised speech signal and the noisy speech signal according to the noise mixing coefficient to obtain a second de-noised speech signal.

[0141] In the illustrative embodiment, step 7031 comprises:

[0142] The noise mixing coefficient is obtained according to the following formula:

[0143]

[0144] Wherein, β is the noise intensity estimation value, A is a preset noise mixing coefficient minimum value, B is a preset noise mixing coefficient maximum value, m is a preset noise intensity filtering range lower limit value, and n is a preset noise intensity filtering range upper limit value.

[0145] Figure 8 is an embodiment schematic diagram of the mapping relationship between the noise mixing coefficient and the noise intensity estimation value according to an illustrative embodiment, as shown in Figure 8 The horizontal axis of the coordinate system is the noise intensity estimation value, wherein the noise intensity estimation value is the signal-to-noise ratio, and the vertical axis of the coordinate system is the noise mixing coefficient, Figure 8 In the embodiment shown, the value range of the noise intensity estimation value is set to [-5db, 15db], so that when the noise intensity estimation value is less than -5db, the noise mixing coefficient is set to 0.01, when the noise intensity estimation value is greater than 15db, the noise mixing coefficient is set to 0.05, and within the [-5db, 15db] interval, the noise mixing coefficient and the noise intensity estimation value are in a linear relationship, that is:

[0146] α = 0.002 * β + 0.02

[0147] In this way, in the illustrative embodiment shown in Figure 8 The change range of the noise mixing coefficient is [0.01, 0.05]. It should be noted that the change range of the noise mixing coefficient is [0.01, 0.05] is only an illustrative example, and in actual application, the change range of the noise mixing coefficient can be adjusted as needed, and the interval [-5db, 15db] of the noise mixing coefficient adjusting with the noise intensity estimation value can also be adjusted as needed.

[0148] In the illustrative embodiment, step 7032 comprises:

[0149] The second de-noised speech signal is obtained by the following formula:

[0150] x_blend = (1-α) * x_denoise + α * x_noise

[0151] Wherein, x blend is the second denoised speech signal, a is the mixed noise coefficient, x denoise is the first denoised speech signal, and x noise is the noisy speech signal.

[0152] Therefore, it can be seen that in the case of low signal-to-noise ratio, the mixed noise coefficient is small, so that the second denoised speech signal contains more first denoised speech signal components and fewer original noisy speech signal components. In the case of high signal-to-noise ratio, the mixed noise coefficient is large, so that the second denoised speech signal contains more original noisy speech signal components and fewer first denoised speech signal components.

[0153] Low signal-to-noise ratio means that the noise is large, so that the first denoised speech signal component contained in the second denoised speech signal can reduce the influence of the noise, and the original noisy speech signal component contained in the second denoised speech signal can make up for the high filtering degree of the pure speech part in the first denoised speech signal, and improve the performance of the pure speech part, so that compared with the first denoised speech signal, the second denoised speech signal improves the listening experience. High signal-to-noise ratio means that the noise is small, so that the first denoised speech signal component contained in the second denoised speech signal can enhance the pure speech part in the original noisy speech signal, and the noise in the original noisy speech signal component contained in the second denoised speech signal is small, and the enhanced pure speech part has a weakening effect on the noise part in the listening experience, so that compared with the first denoised speech signal, the second denoised speech signal improves the listening experience. Therefore, the overall listening experience of the second denoised speech signal is better than that of the first denoised speech signal before mixing.

[0154] Since the GRU network, linear classifier, normalization module, average pooling module and noise type classifier are used in the embodiments of the present disclosure, these structures belong to artificial intelligence neural networks, and need to be trained to achieve the required function and obtain ideal results. Therefore, the embodiments of the present disclosure also include a training process. Figure 9 is a step flow diagram for training a noise reduction classification network model according to an illustrative embodiment. As shown in Figure 9 In the illustrative embodiment, the voice noise reduction processing method of the embodiments of the present disclosure further includes steps 901 to 903.

[0155] Step 901, mixing the pure speech sample signal and the noise sample signal to obtain the noisy speech sample signal.

[0156] In the illustrative embodiment, the noisy speech sample signal is obtained by mixing the pure speech sample signal and the noise sample signal, and the formula is as follows:

[0157]

[0158]

[0159]

[0160] noisy_signal = clean + scalar * noise1

[0161] wherein i is a signal sampling point, N is the total number of sampling points, noise_spe1 represents the power of the noise sample signal, noise1[i] is the signal amplitude of the i-th signal sampling point of the noise sample signal, clean_spe1 represents the power of the clean speech sample signal, clean[i] represents the signal amplitude of the i-th signal sampling point of the clean speech sample signal, snr1 is a preset signal-to-noise ratio, in an illustrative embodiment, snr1 takes a value in the range of [-5db, 15db], scalar is a scaling factor of the noise signal, noisy_signal is the speech signal after mixing with noise, which is used as the input of the deep noise reduction network, clean is the clean speech sample signal, and noise is the noise sample signal.

[0162] In an illustrative embodiment, according to the four categories of steady-state noise, transient noise, human voice noise and other noise, the label 0, 1, 2 and 3 of the noise type is obtained according to the selected noise when the training sample data is made, wherein 0 represents steady-state noise, 1 represents transient noise, 2 represents human voice noise, and 3 represents other noise. In an illustrative embodiment, the sampling rate of the clean speech sample signal is 16000Hz, and 4s of data is randomly cut off for training. If the clean speech sample signal is less than 4s, zero is randomly filled to 4s on the left and right of the clean speech sample signal. In the above formula for mixing the clean speech sample signal and the noise sample signal to obtain the noisy speech sample signal, N = 64000 is obtained according to the sampling rate of 16000Hz and the length of 4s of the clean speech sample signal. In an illustrative embodiment, the length of the noise sample signal is consistent with the length of the clean speech sample signal.

[0163] Step 902, inputting the noisy speech sample signal into a noise reduction classification network model composed of an STFT module, a GRU network, a first linear classifier, a normalization module, a multiplication module, an ISTFT module, a second linear classifier, an average pooling module and a noise type classifier to obtain a noise classification prediction result and a denoised speech prediction signal.

[0164] Step 903, training the noise reduction classification network model according to the difference between the noise classification prediction result and the preset noise classification label, and the difference between the denoised speech prediction signal and the clean speech sample signal, to adjust the parameters in the noise reduction classification network model.

[0165] Figure 10 is a step flowchart diagram illustrating adjusting parameters in a noise reduction classification network model according to a difference during a training process according to an illustrative embodiment, as Figure 10 As shown, step 903 specifically includes steps 1001 to 1004.

[0166] Step 1001, according to the noise classification prediction result and the preset noise classification label, a cross-entropy loss function is established.

[0167] The cross-entropy loss function, i.e. CE Loss (Cross Entropy Loss), has the following formula:

[0168]

[0169] Wherein, ce_loss is the cross-entropy loss, B is the batch sample number, i is the sample serial number, M is the category number, c is the category serial number, y ic is whether the i-th sample is category c, y ic is 0 or 1, p ic is the probability of identifying the i-th sample as category c. In the embodiment of the present disclosure, the cross-entropy loss function represents the difference between the noise classification prediction result and the preset noise classification label.

[0170] Step 1002, according to the noise reduction speech prediction signal and the pure speech sample signal, a scale invariant signal to noise ratio loss function is established.

[0171] The scale invariant signal to noise ratio loss function, i.e. SI-SNR Loss (Scale Invariant Signal to Noise Ratio), has the following formula:

[0172]

[0173] Wherein,

[0174]

[0175]

[0176] Wherein, sisnr_loss is the scale invariant signal to noise ratio loss, s target represents the projection signal of the predicted speech signal on the label speech signal s, s represents the label speech signal, is the speech signal predicted by the model, represents the product sum of the predicted speech signal and the label speech signal corresponding position, and ||s|| 2 represents the power of the label speech signal, and the calculation formula is as follows:

[0177]

[0178]

[0179] wherein, wherein N is the number of sampling points.

[0180] Similarly, ||s target || 2 represents s target the power of the signal, ||e noise || 2 represents e noise the power of the signal, the calculation method is the same.

[0181] In the embodiments of the present disclosure, the scale invariant signal-to-noise ratio loss function characterizes the difference between the denoised speech prediction signal and the clean speech sample signal.

[0182] Step 1003, obtaining the overall loss function of the denoising classification network model according to the cross-entropy loss function and the scale invariant signal-to-noise ratio loss function.

[0183] In the illustrative embodiment, the overall loss function is as follows:

[0184] total_loss = a1*ce_loss + a2*sisnr_loss

[0185] wherein, a1 is the weight of ce_loss, a2 is the weight of sisnr_loss, in the illustrative embodiment, a1 and a2 are both set to 0.5.

[0186] Step 1004, adjusting the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the denoising classification network model through back propagation according to the overall loss function, until the loss value of the overall loss function converges to the expected value or reaches the iteration number of training.

[0187] In the illustrative embodiment, although the denoising classification network model contains the STFT module and the ISTFT module, the STFT and the ISTFT are still in the traditional way, and the parameters used are fixed, so there is no need to train and adjust the parameters.

[0188] The voice noise reduction processing method of the embodiments of the present disclosure performs deep noise reduction processing and classification on the noisy voice signal, and performs adaptive mixing of the voice signal after noise reduction and the original noisy voice signal according to the noise intensity estimation value and the noise category, thereby alleviating the problem of hearing sensation decline caused by excessive noise filtering. Although the voice signal after noise reduction and the original noisy voice signal are mixed to enhance the performance of the noise part, the voice quality and overall audio quality are also improved, so the overall hearing sensation of the voice signal is improved. At the same time, in the voice noise reduction processing method of the embodiments of the present disclosure, the noise reduction classification network model of artificial intelligence is used to complete the prediction of noise reduction and noise type at the same time, thereby enhancing the correlation between the noise reduction structure and the prediction result, and helping to improve the accuracy of the noise type prediction result in judging whether to mix the signals. In addition, in the voice noise reduction processing method of the embodiments of the present disclosure, the noise mixing coefficient changes according to the size of the noise intensity estimation value. In the case of a low noise intensity estimation value, the second noise reduction voice signal retains more first noise reduction voice signal components and fewer original noisy voice signal components. Because the noise component in the original noisy voice signal is high when the noise intensity estimation value is low, the noise reduction process has a greater impact on the pure voice part, so the filtering degree of the pure voice part in the first noise reduction voice signal is high. Therefore, the second noise reduction voice signal retains fewer first noise reduction voice signal components, and more original noisy voice signal components are used to compensate for the high filtering degree of the pure voice part in the first noise reduction voice signal. Although some noise components are introduced, the performance of the voice part is improved in terms of pure voice and overall voice signal, thereby helping to improve the hearing sensation of the noise reduction voice signal, and making the hearing sensation of the second noise reduction voice signal better than that of the first noise reduction voice signal. In the case of a high noise intensity estimation value, the second noise reduction voice signal retains fewer first noise reduction voice signal components and more original noisy voice signal components. Because the noise component in the original noisy voice signal is low when the noise intensity estimation value is high, the noise reduction process has a smaller impact on the pure voice part, so the filtering degree of the pure voice part in the first noise reduction voice signal is low. Therefore, the second noise reduction voice signal retains more first noise reduction voice signal components, and only a small amount of original noisy voice signal components can improve the performance of the pure voice part in the first noise reduction voice signal and the overall performance of the noise reduction voice signal, thereby helping to improve the hearing sensation of the noise reduction voice signal, and making the hearing sensation of the second noise reduction voice signal better than that of the first noise reduction voice signal.

[0189] Figure 11 is a step flowchart diagram of one specific application scenario of the voice noise reduction processing method according to an illustrative embodiment, as shown in Figure 11 the process includes the following steps 1101 to 1114.

[0190] Step 1101, a noise reduction classification network model to be trained is constructed, and then step 1102 is performed.

[0191] The noise reduction classification network model includes an STFT module, a GRU network, a first linear classifier, a normalization module, a multiplication module, an ISTFT module, a second linear classifier, an average pooling module, and a noise type classifier. The structure can be referred to the flow topological diagram of deep noise reduction processing and classification of a noisy speech signal shown in Figure 4 The STFT module is used to perform STFT on the input signal to obtain an amplitude spectrum and a phase spectrum. The GRU network is used to extract features from the amplitude spectrum to obtain audio features. The first linear classifier is used to process the audio features to obtain a first feature vector. The normalization module is used to perform normalization on the first feature vector by using a sigmoid function to obtain an amplitude spectrum mask. The multiplication module is used to multiply the amplitude spectrum and the amplitude spectrum mask to obtain a masked amplitude spectrum. The ISTFT module is used to perform ISTFT on the masked amplitude spectrum and the phase spectrum to obtain a noise reduction signal. The second linear classifier is used to process the audio features to obtain a second feature vector. The average pooling module is used to perform average pooling on the second feature vector to obtain a third feature vector with a time dimension of 1. The noise type classifier is used to obtain a noise class result according to the third feature vector. In the noise reduction classification network model, the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module, and the noise type classifier belong to an artificial intelligence neural network structure, and the weights and other parameters thereof are optimized through training.

[0192] Step 1102, a pure speech sample signal and a noise sample signal are obtained, and the pure speech sample signal and the noise sample signal are mixed to obtain a noisy speech sample signal, and then step 1103 is performed.

[0193] In an illustrative embodiment, the duration of the pure speech sample signal is 4s. If the duration of the speech signal used to make the pure speech sample signal exceeds 4s, the pure speech sample signal is generated by randomly cutting 4s of the speech signal. If the duration of the speech signal is less than 4s, the pure speech sample signal is generated by filling zeros in front of and behind the speech signal to 4s. The duration of the noisy speech sample signal is consistent with that of the pure speech sample signal, which is also 4s. The noisy speech sample signal is obtained by mixing the noise sample signal based on the 4s pure speech sample signal. In an illustrative embodiment, the noise sample signal can be a steady-state noise sample, a transient noise sample, a human voice noise sample, or other noise samples.

[0194] Step 1103, the noisy speech sample signal is input into the noise reduction classification network model to be trained to obtain a noise classification prediction result and a noise reduction speech prediction signal, and then step 1104 is performed.

[0195] Step 1104, according to the noise classification prediction result and the preset noise classification label, a cross entropy loss function is established, according to the noise reduction speech prediction signal and the clean speech sample signal, a scale invariant signal-to-noise ratio loss function is established, and according to the cross entropy loss function and the scale invariant signal-to-noise ratio loss function, the overall loss function of the noise reduction classification network model is obtained, and then step 1105 is executed.

[0196] Step 1105, according to the overall loss function, the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the noise reduction classification network model are adjusted by back propagation, and then step 1106 is executed.

[0197] Step 1106, it is judged whether the loss value of the overall loss function converges to the expected value, if yes, step 1108 is executed, otherwise step 1107 is executed.

[0198] Step 1107, it is judged whether the iteration number of training is reached, if yes, step 1108 is executed, otherwise step 1102 is executed.

[0199] Step 1108, the noisy speech signal is obtained, the noisy speech signal is input into the trained noise reduction classification network model, the first noise reduction speech signal and the noise category are obtained, and then step 1109 is executed.

[0200] Step 1109, according to the first noise reduction speech signal and the noisy speech signal, the noise signal estimation value in the noisy speech signal is obtained, and then step 1110 is executed.

[0201] Step 1110, according to the noise signal estimation value and the first noise reduction speech signal respectively, the noise signal power and the first noise reduction speech signal power are obtained, and then step 1111 is executed.

[0202] Step 1111, according to the noise signal power and the first noise reduction speech signal power, the signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power is obtained, and then step 1112 is executed.

[0203] In an illustrative embodiment, the signal-to-noise ratio is obtained by the following formula, that is, the noise intensity estimation value:

[0204]

[0205] Wherein, snr is the signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power, clean_spe2 is the first noise reduction speech signal power, and noise_spe2 is the noise signal power.

[0206] Step 1112, judging whether the noise category is steady-state noise, if yes, executing step 1113, otherwise ending the flow.

[0207] Step 1113, obtaining a noise mixing coefficient according to a signal-to-noise ratio between the noise signal power and the first denoised speech signal power, and then executing step 1114.

[0208] In an illustrative embodiment, the noise mixing coefficient is obtained according to the following formula:

[0209]

[0210] wherein β is a signal-to-noise ratio (noise intensity estimation value) between the noise signal power and the first denoised speech signal power, i.e. snr in step 1111, A is a preset lowest value of the noise mixing coefficient, B is a preset highest value of the noise mixing coefficient, m is a preset lower limit value of the noise intensity screening range, and n is a preset upper limit value of the noise intensity screening range.

[0211] Step 1114, mixing the first denoised speech signal and the noisy speech signal according to the noise mixing coefficient to obtain a second denoised speech signal.

[0212] In an illustrative embodiment, the second denoised speech signal is obtained by the following formula:

[0213] x_blend = (1 - a) * x_denoise + a * x_noise

[0214] wherein x_blend is the second denoised speech signal, a is the noise mixing coefficient, x_denoise is the first denoised speech signal, and x_noise is the noisy speech signal.

[0215] At this point, the application scenario flow is completed.

[0216] Figure 12 is a logic structure schematic diagram of a speech denoising processing device according to an illustrative embodiment, as shown in Figure 12 The speech denoising processing device includes a signal acquisition module 1201, a denoising classification module 1202, a noise intensity obtaining module 1203, and a denoised speech obtaining module 1204.

[0217] The signal acquisition module 1201 is configured to execute acquisition of a noisy speech signal.

[0218] The denoising classification module 1202 is configured to execute deep denoising processing and classification of the noisy speech signal to obtain a first denoised speech signal and a noise category.

[0219] The noise intensity obtaining module 1203 is configured to execute obtaining of a noise intensity estimation value according to the first denoised speech signal and the noisy speech signal.

[0220] The noise-reduced speech obtaining module 1204 is configured to perform signal mixing on the first noise-reduced speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise-reduced speech signal.

[0221] In an illustrative embodiment, the noise-reduction classification module 1202 includes:

[0222] The STFT unit is configured to perform inputting the noisy speech signal into a short-time Fourier transform (STFT) module, and performing STFT on the noisy speech signal by the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum.

[0223] The feature extraction unit is configured to perform inputting the noisy speech amplitude spectrum into a GRU network, and performing feature extraction on the noisy speech amplitude spectrum by the GRU network to obtain noisy speech features.

[0224] The first feature obtaining unit is configured to perform inputting the noisy speech features into a first linear classifier, and obtaining a first feature vector by the first linear classifier.

[0225] The mask obtaining unit is configured to perform inputting the first feature vector into a normalization module, and obtaining an amplitude spectrum mask by the normalization module.

[0226] The masked audio amplitude spectrum obtaining unit is configured to perform inputting the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module, and multiplying the noisy speech amplitude spectrum and the amplitude spectrum mask by the multiplication module to obtain a masked audio amplitude spectrum.

[0227] The ISTFT unit is configured to perform inputting the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module, and performing ISTFT on the masked audio amplitude spectrum and the noisy speech phase spectrum by the ISTFT module to obtain the first noise-reduced speech signal.

[0228] The second feature obtaining unit is configured to perform inputting the noisy speech features into a second linear classifier, and obtaining a second feature vector by the second linear classifier.

[0229] The pooling unit is configured to perform inputting the second feature vector into an average pooling (avg_pooling) module, and performing average pooling on the second feature vector by the average pooling module to obtain a third feature vector.

[0230] The classification unit is configured to perform inputting the third feature vector into a noise type classifier, and obtaining the noise category by the noise type classifier.

[0231] In an illustrative embodiment, the speech noise-reduction processing apparatus further includes:

[0232] a sample mixing module configured to perform mixing a clean speech sample signal and a noise sample signal to obtain a noisy speech sample signal;

[0233] a prediction result obtaining module configured to perform inputting the noisy speech sample signal into a noise reduction classification network model composed of an STFT module, a GRU network, a first linear classifier, a normalization module, a multiplication module, an ISTFT module, a second linear classifier, an average pooling module, and a noise type classifier to obtain a noise classification prediction result and a noise reduced speech prediction signal;

[0234] a model training module configured to perform training the noise reduction classification network model according to a difference between the noise classification prediction result and a preset noise classification label, and a difference between the noise reduced speech prediction signal and the clean speech sample signal to adjust parameters in the noise reduction classification network model.

[0235] In an illustrative embodiment, the model training module includes:

[0236] a first loss function establishing submodule configured to perform establishing a cross-entropy loss function according to the noise classification prediction result and the preset noise classification label;

[0237] a second loss function establishing submodule configured to perform establishing a scale-invariant signal-to-noise ratio loss function according to the noise reduced speech prediction signal and the clean speech sample signal;

[0238] a total loss function obtaining submodule configured to perform obtaining a total loss function of the noise reduction classification network model according to the cross-entropy loss function and the scale-invariant signal-to-noise ratio loss function;

[0239] a model parameter adjusting submodule configured to perform adjusting parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module, and the noise type classifier in the noise reduction classification network model by back propagation according to the total loss function until a loss value of the total loss function converges to an expected value or reaches an iteration number of training.

[0240] In an illustrative embodiment, the noise intensity obtaining module 1203 includes:

[0241] a noise signal obtaining submodule configured to perform obtaining a noise signal estimation value in the noisy speech signal according to the first noise reduced speech signal and the noisy speech signal;

[0242] a signal power obtaining submodule configured to perform obtaining a noise signal power and a first noise reduced speech signal power according to the noise signal estimation value and the first noise reduced speech signal, respectively;

[0243] The noise intensity obtaining sub-module is configured to obtain a noise intensity estimation value according to the noise signal power and the first noise-reduced speech signal power.

[0244] The noise intensity obtaining sub-module is further configured to determine a signal-to-noise ratio between the noise signal power and the first noise-reduced speech signal power as the noise intensity estimation value.

[0245] In an illustrative embodiment, the noise-reduced speech obtaining module 1204 is further configured to:

[0246] determine whether signal mixing of the first noise-reduced speech signal and the noisy speech signal is needed according to the noise category;

[0247] in a case where the signal mixing of the first noise-reduced speech signal and the noisy speech signal is not needed, not perform the signal mixing of the first noise-reduced speech signal and the noisy speech signal;

[0248] in a case where the signal mixing of the first noise-reduced speech signal and the noisy speech signal is needed, perform the following steps:

[0249] obtain a noise mixing coefficient according to the noise intensity estimation value;

[0250] mix the first noise-reduced speech signal and the noisy speech signal according to the noise mixing coefficient to obtain a second noise-reduced speech signal.

[0251] In an illustrative embodiment, the noise category includes steady-state noise and transient noise; and the noise-reduced speech obtaining module 1204 is further configured to:

[0252] in a case where the noise category is steady-state noise, determine that the signal mixing of the first noise-reduced speech signal and the noisy speech signal is needed; or

[0253] in a case where the noise category is not steady-state noise, determine that the signal mixing of the first noise-reduced speech signal and the noisy speech signal is not needed.

[0254] In an illustrative embodiment, obtaining the noise mixing coefficient according to the noise intensity estimation value includes:

[0255] obtaining the noise mixing coefficient according to the following formula:

[0256]

[0257] wherein β is the noise intensity estimation value, A is a preset lowest value of the noise mixing coefficient, B is a preset highest value of the noise mixing coefficient, m is a preset lower limit value of a noise intensity screening range, and n is a preset upper limit value of the noise intensity screening range.

[0258] In the illustrative embodiment, mixing the first de-noised speech signal and the noisy speech signal according to a noise mixing coefficient to obtain a second de-noised speech signal, comprising:

[0259] The second de-noised speech signal is obtained by the following formula:

[0260] x_blend = (1 - a) * x_denoise + a * x_noise

[0261] Wherein, x_blend is the second de-noised speech signal, a is the noise mixing coefficient, x_denoise is the first de-noised speech signal, and x_noise is the noisy speech signal.

[0262] The voice noise reduction processing device of the embodiments of the present disclosure performs deep noise reduction processing and classification on the noisy voice signal, and performs adaptive mixing of the voice signal after noise reduction and the original noisy voice signal according to the noise intensity estimation value and the noise category, thereby alleviating the problem of hearing sensation decline caused by excessive noise filtering. Although the voice signal after noise reduction and the original noisy voice signal are mixed to enhance the performance of the noise part, the voice quality and overall audio quality are also improved, so the overall hearing sensation of the voice signal is improved. At the same time, in the voice noise reduction processing device of the embodiments of the present disclosure, the noise reduction classification network model of artificial intelligence simultaneously completes noise reduction and noise type prediction, thereby enhancing the correlation between the noise reduction structure and the prediction result, and helping to improve the accuracy of the noise type prediction result in judging whether to mix the signals. In addition, in the voice noise reduction processing device of the embodiments of the present disclosure, the noise mixing coefficient changes according to the size of the noise intensity estimation value. In the case of a low noise intensity estimation value, the second noise reduction voice signal retains more first noise reduction voice signal components and fewer original noisy voice signal components. Because the noise component in the original noisy voice signal is high when the noise intensity estimation value is low, the noise reduction process has a greater impact on the pure voice part, so the filtering degree of the pure voice part in the first noise reduction voice signal is high. Therefore, the second noise reduction voice signal retains fewer first noise reduction voice signal components, and makes up for the high filtering degree of the pure voice part in the first noise reduction voice signal by using more original noisy voice signal components. Although some noise components are introduced, the performance of the voice part is improved in terms of pure voice and overall voice signal, thereby helping to improve the hearing sensation of the noise reduction voice signal, and making the hearing sensation of the second noise reduction voice signal better than that of the first noise reduction voice signal. In the case of a high noise intensity estimation value, the second noise reduction voice signal retains fewer first noise reduction voice signal components and more original noisy voice signal components. Because the noise component in the original noisy voice signal is low when the noise intensity estimation value is high, the noise reduction process has a smaller impact on the pure voice part, so the filtering degree of the pure voice part in the first noise reduction voice signal is low. Therefore, the second noise reduction voice signal retains more first noise reduction voice signal components, and only uses fewer original noisy voice signal components to improve the performance of the pure voice part in the first noise reduction voice signal and the overall performance of the noise reduction voice signal, thereby helping to improve the hearing sensation of the noise reduction voice signal, and making the hearing sensation of the second noise reduction voice signal better than that of the first noise reduction voice signal.

[0263] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described here.

[0264] As to the voice noise reduction processing apparatus in the above embodiments, the specific manners in which the units perform operations have been described in detail in the embodiments of the voice noise reduction processing method, and thus will not be described in detail here.

[0265] It should be noted that the above embodiments are only used for example to divide the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions.

[0266] Figure 13 FIG. 13 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. In some embodiments, the electronic device is a server. The electronic device 1300 can be quite different due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1301 and one or more memories 1302, wherein the memory 1302 stores at least one program code, the at least one program code is loaded and executed by the processor 1301 to implement the voice noise reduction processing method provided by each of the above embodiments. Of course, the electronic device 1300 can also have a wired or wireless network interface, a keyboard, and an input and output interface, etc., so as to perform input and output, and the electronic device 1300 can also include other components for realizing the functions of the device, which will not be described here.

[0267] The above only describes preferred embodiments of the present disclosure and is not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A speech noise reduction processing method, comprising: obtaining a noisy speech signal; performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category; obtaining a noise intensity estimate value according to the first noise-reduced speech signal and the noisy speech signal; performing signal mixing on the first noise-reduced speech signal and the noisy speech signal according to the noise intensity estimate value and the noise category to obtain a second noise-reduced speech signal; wherein the deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category comprises: inputting the noisy speech signal into a short-time Fourier transform (STFT) module, performing STFT on the noisy speech signal by the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum; inputting the noisy speech amplitude spectrum into a GRU network, performing feature extraction on the noisy speech amplitude spectrum by the GRU network to obtain a noisy speech feature; inputting the noisy speech feature into a first linear classifier to obtain a first feature vector by the first linear classifier; inputting the first feature vector into a normalization module to obtain an amplitude spectrum mask by the normalization module; inputting the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module to multiply the noisy speech amplitude spectrum and the amplitude spectrum mask to obtain a masked audio amplitude spectrum by the multiplication module; inputting the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module, performing ISTFT on the masked audio amplitude spectrum and the noisy speech phase spectrum by the ISTFT module to obtain the first noise-reduced speech signal; inputting the noisy speech feature into a second linear classifier to obtain a second feature vector by the second linear classifier; inputting the second feature vector into an average pooling module to perform average pooling on the second feature vector by the average pooling module to obtain a third feature vector; inputting the third feature vector into a noise type classifier to obtain the noise category by the noise type classifier.

2. The voice de-noising processing method of claim 1, wherein, The speech noise reduction processing method further comprises: mixing a clean speech sample signal and a noise sample signal to obtain a noisy speech sample signal; inputting the noisy speech sample signal into a noise reduction classification network model composed of the STFT module, the GRU network, the first linear classifier, the normalization module, the multiplication module, the ISTFT module, the second linear classifier, the average pooling module, and the noise type classifier to obtain a noise classification prediction result and a noise-reduced speech prediction signal; training the noise reduction classification network model according to the difference between the noise classification prediction result and a preset noise classification label, and the difference between the noise-reduced speech prediction signal and the clean speech sample signal to adjust the parameters in the noise reduction classification network model.

3. The voice de-noising processing method of claim 2, wherein, The difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the pure speech sample signal are used to train the noise reduction classification network model, so as to adjust parameters in the noise reduction classification network model, including: According to the noise classification prediction result and the preset noise classification label, a cross-entropy loss function is established; According to the noise reduction speech prediction signal and the pure speech sample signal, a scale-invariant signal-to-noise ratio loss function is established; According to the cross-entropy loss function and the scale-invariant signal-to-noise ratio loss function, a total loss function about the noise reduction classification network model is obtained; According to the total loss function, the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the noise reduction classification network model are adjusted through back propagation until the loss value of the total loss function converges to an expected value or reaches a training iteration number.

4. The voice de-noising processing method of claim 1, wherein, The noise intensity estimation value is obtained according to the first noise reduction speech signal and the noisy speech signal, including: According to the first noise reduction speech signal and the noisy speech signal, a noise signal estimation value in the noisy speech signal is obtained; According to the noise signal estimation value and the first noise reduction speech signal respectively, noise signal power and first noise reduction speech signal power are obtained; According to the noise signal power and the first noise reduction speech signal power, the noise intensity estimation value is obtained.

5. The voice de-noising processing method of claim 4, wherein, The noise intensity estimation value is obtained according to the noise signal power and the first noise reduction speech signal power, including: The signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power is determined as the noise intensity estimation value.

6. The voice de-noising processing method of claim 1, wherein, The first noise reduction speech signal and the noisy speech signal are mixed according to the noise intensity estimation value and the noise category to obtain a second noise reduction speech signal, including: According to the noise category, it is determined whether the first noise reduction speech signal and the noisy speech signal need to be mixed; In the case that the first noise reduction speech signal and the noisy speech signal do not need to be mixed, the first noise reduction speech signal and the noisy speech signal are not mixed; or In the case that the first noise reduction speech signal and the noisy speech signal need to be mixed, the following steps are performed: According to the noise intensity estimation value, a noise mixing coefficient is obtained; The first noise reduction speech signal and the noisy speech signal are mixed according to the noise mixing coefficient to obtain the second noise reduction speech signal.

7. The speech noise reduction processing method according to claim 6, characterized in that: The noise category at least includes steady-state noise; According to the noise category, it is determined whether the first noise reduction speech signal and the noisy speech signal need to be mixed, including: In the case that the noise category is the steady-state noise, it is determined that the first noise reduction speech signal and the noisy speech signal need to be mixed; or In a case where the noise category is not the stationary noise, it is determined that signal mixing of the first de-noised speech signal and the noisy speech signal is not required.

8. The voice de-noising processing method of claim 6, wherein, The mixing noise coefficient is obtained according to the noise intensity estimation value, including: The mixing noise coefficient is obtained according to the noise intensity estimation value, including: The mixing noise coefficient is obtained according to the noise intensity estimation value, including:

9. The voice de-noising processing method of claim 6, wherein, Wherein, β is the noise intensity estimation value, A is the preset lowest value of the mixing noise coefficient, B is the preset highest value of the mixing noise coefficient, m is the preset lower limit value of the noise intensity screening range, and n is the preset upper limit value of the noise intensity screening range. The mixing noise coefficient is obtained according to the noise intensity estimation value, including: The second de-noised speech signal is obtained by mixing the first de-noised speech signal and the noisy speech signal according to the mixing noise coefficient, including: The second de-noised speech signal is obtained by mixing the first de-noised speech signal and the noisy speech signal according to the mixing noise coefficient, including:

10. A speech noise reduction processing apparatus characterized by comprising: Wherein, x_blend is the second de-noised speech signal, α is the mixing noise coefficient, x_denoise is the first de-noised speech signal, and x_noise is the noisy speech signal. Including: The signal acquisition module is configured to perform the acquisition of the noisy speech signal; The noise classification module is configured to perform deep de-noising processing and classification on the noisy speech signal to obtain a first de-noised speech signal and a noise category; The noise intensity obtaining module is configured to obtain a noise intensity estimation value according to the first de-noised speech signal and the noisy speech signal; The de-noised speech obtaining module is configured to perform signal mixing of the first de-noised speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second de-noised speech signal; The noise classification module includes: The STFT unit is configured to input the noisy speech signal into a short-time Fourier transform (STFT) module, and perform STFT on the noisy speech signal through the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum; The feature extraction unit is configured to input the noisy speech amplitude spectrum into a GRU network, and perform feature extraction on the noisy speech amplitude spectrum through the GRU network to obtain a noisy speech feature; The first feature acquisition unit is configured to input the noisy speech feature into a first linear classifier, and obtain a first feature vector through the first linear classifier; The mask acquisition unit is configured to input the first feature vector into a normalization module, and obtain an amplitude spectrum mask through the normalization module; The masked audio amplitude spectrum acquisition unit is configured to input the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module, and obtain a masked audio amplitude spectrum by multiplying the noisy speech amplitude spectrum and the amplitude spectrum mask through the multiplication module; The ISTFT unit is configured to input the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module, and perform ISTFT on the input of the masked audio amplitude spectrum and the noisy speech phase spectrum through the ISTFT module to obtain the first de-noised speech signal; The second feature acquisition unit is configured to input the noisy speech feature into a second linear classifier, and obtain a second feature vector through the second linear classifier. The pooling unit is configured to input the second feature vector into an average pooling module, perform average pooling on the second feature vector through the average pooling module, and obtain a third feature vector. The classification unit is configured to input the third feature vector into a noise type classifier, and obtain the noise category through the noise type classifier.

Images