Voice noise reduction processing method and device

Abstract

The invention relates to a voice noise reduction processing method and device, and the method comprises the steps: obtaining a noisy voice signal; performing deep noise reduction processing and classification on the noisy voice signal to obtain a first noise-reduced voice signal and a noise category; obtaining a noise intensity estimation value according to the first noise reduction voice signal and the noisy voice signal; and according to the noise intensity estimation value and the noise type, performing signal mixing on the first noise reduction voice signal and the noise-containing voice signal to obtain a second noise reduction voice signal. According to the second noise reduction voice signal obtained by the technical scheme of the invention, the problem that the hearing feeling is reduced due to the excessive filtering of the noise is alleviated, the voice quality and the overall audio quality are improved, and the hearing feeling of the voice signal is improved on the whole.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to a method and device for speech noise reduction processing. Background Art

[0002] Speech noise reduction refers to the technology of removing unnecessary noise from sound signals to improve speech quality. In real life, voice communication or speech recognition technology is often used due to the complex and changeable environment, such as conference rooms, transportation, streets and other scenes. These environments usually contain various noises, such as wind noise, car noise, machine noise, human voice noise, etc. These noises will affect the quality of speech signals, not only affecting the listening experience, but also causing a decrease in the accuracy of speech recognition.

[0003] To solve this problem, speech noise reduction technology has emerged. However, in existing speech noise reduction technology, while filtering out noise, the speech that needs to be retained will also be affected, resulting in poor listening quality of the final noise-reduced speech. Summary of the invention

[0004] In view of this, the present disclosure provides a method and device for speech noise reduction processing to improve the effect of speech noise reduction processing and improve the listening experience of the noise-reduced speech signal.

[0005] The technical solution of the present disclosure is achieved as follows:

[0006] A speech noise reduction processing method, comprising:

[0007] Acquire a noisy speech signal;

[0008] Performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category;

[0009] Obtaining a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal;

[0010] According to the noise intensity estimation value and the noise category, the first noise-reduced speech signal and the noisy speech signal are mixed to obtain a second noise-reduced speech signal.

[0011] Further, the performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise reduction speech signal and a noise category includes:

[0012] Inputting the noisy speech signal into a short-time Fourier transform (STFT) module, and performing STFT on the noisy speech signal by the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum;

[0013] Inputting the noisy speech amplitude spectrum into a GRU network, and performing feature extraction on the noisy speech amplitude spectrum through the GRU network to obtain noisy speech features;

[0014] Inputting the noisy speech feature into a first linear classifier, and obtaining a first feature vector through the first linear classifier;

[0015] Inputting the first feature vector into a normalization module, and obtaining an amplitude spectrum mask through the normalization module;

[0016] Inputting the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module, and multiplying the noisy speech amplitude spectrum and the amplitude spectrum mask by the multiplication module to obtain a masked audio amplitude spectrum;

[0017] Input the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module, and perform ISTFT on the masked audio amplitude spectrum and the noisy speech phase spectrum input by the ISTFT module to obtain the first noise reduction speech signal;

[0018] Inputting the noisy speech feature into a second linear classifier, and obtaining a second feature vector through the second linear classifier;

[0019] Inputting the second feature vector into an average pooling module, and performing average pooling on the second feature vector through the average pooling module to obtain a third feature vector;

[0020] The third feature vector is input into a noise type classifier, and the noise category is obtained through the noise type classifier.

[0021] Furthermore, the speech noise reduction processing method further includes:

[0022] The clean speech sample signal and the noise sample signal are mixed to obtain a noisy speech sample signal;

[0023] Inputting the noisy speech sample signal into a noise reduction classification network model composed of the STFT module, the GRU network, the first linear classifier, the normalization module, the multiplication module, the ISTFT module, the second linear classifier, the average pooling module and the noise type classifier to obtain a noise classification prediction result and a noise reduction speech prediction signal;

[0024] According to the difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the clean speech sample signal, the noise reduction classification network model is trained to adjust the parameters in the noise reduction classification network model.

[0025] Further, the noise reduction classification network model is trained according to the difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the clean speech sample signal to adjust the parameters in the noise reduction classification network model, including:

[0026] Establishing a cross entropy loss function according to the noise classification prediction result and the preset noise classification label;

[0027] Establishing a scale-invariant signal-to-noise ratio loss function according to the noise-reduced speech prediction signal and the clean speech sample signal;

[0028] According to the cross entropy loss function and the scale-invariant signal-to-noise ratio loss function, an overall loss function of the denoising classification network model is obtained;

[0029] According to the overall loss function, the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the denoising classification network model are adjusted through back propagation until the loss value of the overall loss function converges to the expected value or reaches the number of training iterations.

[0030] Further, obtaining a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal includes:

[0031] Obtaining a noise signal estimation value in the noisy speech signal according to the first noise reduction speech signal and the noisy speech signal;

[0032] Obtaining noise signal power and first noise reduction speech signal power according to the noise signal estimation value and the first noise reduction speech signal respectively;

[0033] The noise intensity estimation value is obtained according to the noise signal power and the first noise reduction speech signal power.

[0034] Further, obtaining the noise intensity estimation value according to the noise signal power and the first noise reduction speech signal power includes:

[0035] The signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power is determined as the noise intensity estimation value.

[0036] Further, the mixing of the first noise reduction speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise reduction speech signal includes:

[0037] Determining whether it is necessary to perform signal mixing on the first noise reduction speech signal and the noisy speech signal according to the noise category;

[0038] When there is no need to mix the first noise reduction speech signal and the noisy speech signal, do not mix the first noise reduction speech signal and the noisy speech signal; or

[0039] In the case where the first noise reduction speech signal and the noisy speech signal need to be mixed, the following steps are performed:

[0040] Obtaining a noise mixing coefficient according to the noise intensity estimation value;

[0041] The first noise-reduced speech signal and the noisy speech signal are mixed according to the noise mixing coefficient to obtain the second noise-reduced speech signal.

[0042] Further, the noise category includes at least steady-state noise;

[0043] The determining, according to the noise category, whether it is necessary to perform signal mixing on the first noise reduction speech signal and the noisy speech signal comprises:

[0044] When the noise category is the steady-state noise, determining that the first noise reduction speech signal and the noisy speech signal need to be mixed; or

[0045] When the noise category is not the steady-state noise, it is determined that there is no need to perform signal mixing on the first noise reduction speech signal and the noisy speech signal.

[0046] Further, obtaining a noise mixing coefficient according to the noise intensity estimation value includes:

[0047] The noise mixing coefficient is obtained according to the following formula:

[0048]

[0049] Wherein, β is the noise intensity estimation value, A is the preset minimum value of the mixed noise coefficient, B is the preset maximum value of the mixed noise coefficient, m is the preset lower limit value of the noise intensity screening range, and n is the preset upper limit value of the noise intensity screening range.

[0050] Further, the mixing the first noise-reduced speech signal and the noisy speech signal according to the noise mixing coefficient to obtain the second noise-reduced speech signal includes:

[0051] The second noise reduction speech signal is obtained by the following formula:

[0052] x_blend=(1-α)*x_denoise+α*x_noise

[0053] Among them, x_blend is the second denoised speech signal, α is the mixing noise coefficient, x_denoise is the first denoised speech signal, and x_noise is the noisy speech signal.

[0054] A speech noise reduction processing device, comprising:

[0055] A signal acquisition module, configured to acquire a noisy speech signal;

[0056] A noise reduction classification module is configured to perform deep noise reduction processing and classification on the noisy speech signal to obtain a first noise reduction speech signal and a noise category;

[0057] A noise intensity obtaining module, configured to obtain a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal;

[0058] The noise reduction speech acquisition module is configured to perform signal mixing on the first noise reduction speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise reduction speech signal.

[0059] It can be seen from the above scheme that the speech noise reduction processing method and device disclosed in the present invention perform deep noise reduction processing and classification on the noisy speech signal, and adaptively mix the noise-reduced speech signal with the original noisy speech signal according to the noise intensity estimation value and the noise category, thereby alleviating the problem of decreased hearing caused by excessive noise filtering. Although the noise-reduced speech signal is mixed with the original noisy speech signal to enhance the performance of the noise part, the speech quality and the overall audio quality are also improved at the same time, thus improving the hearing of the speech signal as a whole. At the same time, the speech noise reduction processing method and device disclosed in the present invention use the artificial intelligence noise reduction classification network model to simultaneously complete the prediction of noise reduction and noise type, thereby enhancing the correlation between the noise reduction structure and the prediction results, and helping to improve the accuracy of the noise type prediction results in judging whether to perform signal mixing. Moreover, in the speech noise reduction processing method and device disclosed in the present invention, the mixing noise coefficient varies according to the size of the noise intensity estimation value. When the noise intensity estimation value is low, the second noise reduction speech signal retains more first noise reduction speech signal components and fewer original noisy speech signal components. Because the noise component in the original noisy speech signal is high when the noise intensity estimation value is low, the noise reduction process has a greater impact on the pure speech part, resulting in a higher degree of filtering of the pure speech part in the first noise reduction speech signal. Therefore, the second noise reduction speech signal retains fewer first noise reduction speech signal components, and more original noisy speech signal components are used to compensate for the pure speech part with a higher degree of filtering in the first noise reduction speech signal. Although a certain amount of noise components are introduced, the compensation for the pure speech part with a higher degree of filtering improves the performance of the speech part from both the pure speech perspective and the overall perspective of the speech signal, thereby helping to improve the noise reduction speech signal. The second denoised speech signal has a better hearing experience than the first denoised speech signal; when the noise intensity estimation value is high, the second denoised speech signal retains fewer first denoised speech signal components and more original noisy speech signal components. Because the noise component in the original noisy speech signal is low when the noise intensity estimation value is high, the noise reduction process has less impact on the pure speech part, so that the degree of filtering of the pure speech part in the first denoised speech signal is low, so the second denoised speech signal retains more first denoised speech signal components. Only with fewer original noisy speech signal components, the performance of the pure speech part in the first denoised speech signal and the overall performance of the denoised speech signal can be improved, thereby helping to improve the hearing experience of the denoised speech signal, so that the hearing experience of the second denoised speech signal is better than the first denoised speech signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of a method for speech noise reduction processing according to an exemplary embodiment;

[0061] Figure 2 is a modular flow chart of a speech noise reduction processing method according to an exemplary embodiment;

[0062] Figure 3 is a schematic diagram of the steps of performing deep noise reduction processing and classification on a noisy speech signal according to an exemplary embodiment;

[0063] Figure 4 is a schematic diagram of a process topology for performing deep noise reduction processing and classification on a noisy speech signal according to an exemplary embodiment;

[0064] Figure 5 is a schematic diagram of the logical structure of a GRU unit in a GRU network according to an exemplary embodiment

[0065] Figure 6 is a schematic flow chart of steps for obtaining a noise intensity estimation value according to an exemplary embodiment;

[0066] Figure 7 is a schematic flow chart of steps for mixing a first noise reduction speech signal and a noisy speech signal according to an exemplary embodiment;

[0067] Figure 8 is a schematic diagram of an embodiment of a mapping relationship between a noise mixing coefficient and a noise intensity estimation value according to an exemplary embodiment;

[0068] Fig. 9 is a schematic flow chart of steps for training a denoising classification network model according to an exemplary embodiment;

[0069] Fig.10 is a schematic flow chart of steps for adjusting parameters in a denoising classification network model according to differences during a training process according to an exemplary embodiment;

[0070] Fig.11 is a schematic diagram of a step flow chart of a specific application scenario of a speech noise reduction processing method according to an exemplary embodiment;

[0071] Fig.12 is a logical structure diagram of a speech noise reduction processing device according to an exemplary embodiment;

[0072] Fig.13 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and examples.

[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] Speech noise reduction can use traditional digital signal processing technology to remove noise, including filters, spectral subtraction, statistical models and other methods. With the development of deep learning in recent years, deep learning algorithms are mostly used to process noisy data and extract pure speech signals. Traditional digital signal processing technology has a good effect on removing steady-state noise (such as long-lasting wind noise, air conditioning noise, car running noise, etc.), but it is not effective in filtering transient noise (such as door closing, whistle, thunder and lightning, etc.). Deep learning can filter steady-state noise and transient noise well, but deep learning will over-filter noise in scenarios with low signal-to-noise ratio and high noise intensity, causing the audio after noise reduction to lose some high-frequency or low-frequency information, which in turn leads to poor listening quality of the noise-reduced speech.

[0076] In view of this, the embodiments of the present disclosure provide a speech noise reduction processing method and device, which adopts a deep learning method to reduce the noise of the speech signal, and simultaneously predicts the noise type in the neural network structure, and divides the noise type into four categories: steady-state noise, transient noise, human voice noise, and other noise. According to the noise type and noise intensity, the denoised speech signal is further processed to improve the effect of the noise reduction audio, and is mixed with the original speech signal to improve the hearing of the noise reduction audio. The technical solution of the embodiments of the present disclosure can be applied to terminal devices with voice pickup, such as surveillance cameras and TWS (True Wireless Stereo) headphones.

[0077] Figure 1 is a flow chart of a method for speech noise reduction processing according to an exemplary embodiment. Figure 2 FIG. 1 is a modular flow chart of a method for speech noise reduction processing according to an exemplary embodiment. Figure 1 Combined with Figure 2 As shown, the speech noise reduction processing method mainly includes the following steps 101 to 104.

[0078] Step 101: Acquire a noisy speech signal.

[0079] In the exemplary embodiment, the noisy speech signal is a speech signal containing background noise, such as speech sounds collected by an audio collection device. Since most people are in a non-quiet environment, when collecting speech sounds, the audio collection device will also collect background noise in the environment, such as wind, rain, thunder, and vehicle driving in outdoor environments, background human speech, walking sounds, and the movement of objects such as tables and chairs in a noisy indoor environment, etc. When making a voice call or video conference, these background noises will be collected by the audio collection device together with the speech signal, thereby obtaining a noisy speech signal.

[0080] Step 102: perform deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category.

[0081] In an exemplary embodiment, step 102 is implemented using STFT (Short Time Fourier Transform), neural network reasoning, and ISTFT (Inverse Short Time Fourier Transform). Figure 3 is a schematic diagram of the steps of performing deep noise reduction processing and classification on a noisy speech signal according to an exemplary embodiment, Figure 4 FIG. 1 is a schematic diagram of a process topology for performing deep noise reduction processing and classification on a noisy speech signal according to an exemplary embodiment. Figure 3 , Figure 4 As shown, step 102 mainly includes the following steps 301 to 309.

[0082] Step 301: Input a noisy speech signal into an STFT module, and perform STFT on the noisy speech signal through the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum.

[0083] In an illustrative embodiment, the sampling rate of the noisy speech signal is 16000 Hz, and the dimension of the noisy speech signal is dim=(32, 1, 64000), where 32 is the batch_size, 1 is the number of channels, 64000 is the number of speech sampling points (with a sampling rate of 16000 Hz and a time length of 4 seconds as a reference), and batch_size represents the size of batch processing data.

[0084] In an exemplary embodiment, the parameters of the STFT are, for example: nfft=320, hop_length=160, win_length=320, where nfft is the number of Fourier points, hop_length is the step length, and win_length is the window length.

[0085] Among them, STFT is a method for analyzing time domain signals into time-frequency diagrams. STFT realizes spectrum analysis by dividing the signal into multiple time windows and performing fast Fourier transform on each window. In STFT, the number of points refers to the number of discrete sampling points used in each window, the step size refers to the time interval for analysis on the input signal, and the window length refers to the length of the time window used when analyzing the signal.

[0086] In the exemplary embodiment, the dimensions of the amplitude spectrum of noisy speech and the phase spectrum of noisy speech are both dim=(32, 161, 400), where 32 is the batch size (batch_size), 161 is the number of channels, and 400 is the time length. Since the above step size is set to 160 points, that is, 10ms per frame, the time length of the 4s speech signal is 400. The batch size determines the amount of data contained in each batch of data input to the GRU network. When training the neural network, the batch size can be adjusted according to the size of the neural network model and the GPU (Graphic Process Unit) memory used. For Fourier transform, because the Fourier transform obtains a conjugate complex number, the amplitude spectrum is left-right symmetrical, so the number of effective feature points is nfft / 2+1, and because the number of Fourier points is nfft=320, the number of effective feature points is 320 / 2+1=161, that is, the value of the number of channels.

[0087] Step 302: input the noisy speech amplitude spectrum into the GRU network, and extract features of the noisy speech amplitude spectrum through the GRU network to obtain noisy speech features.

[0088] Among them, GRU, namely Gated Recurrent Unit, is a variant of the Recurrent Neural Network (RNN) that can be used to process sequence data and model time dependencies.

[0089] Figure 5 is a schematic diagram of the logical structure of a GRU unit in a GRU network according to an exemplary embodiment, such as Figure 5 As shown, the GRU unit uses the update gate and reset gate as the core modules, and the input variable x t The state memory variable h of the previous moment t-1 The concatenated matrix is ​​transformed by sigmoid nonlinearity and then input into the update gate to determine the extent to which the state variables at the previous moment are brought into the current state. The reset gate controls the amount of information written into the candidate set at the previous moment, through Iz t times h t-1 Store the information of the previous moment, through z t times Record the information at the current moment, add the information at the previous moment and the information at the current moment as the output h at the current moment t .

[0090] The mathematical description formula of the GRU unit is as follows:

[0091]

[0092] Among them, x t is the input vector, h t-1 is the state memory variable of the previous moment, h t is the state memory variable at the current moment, r t To update the state of the door, z t To reset the state of the door, is the state of the current candidate set, y t is the output vector at the current moment, W r is the weight parameter of the update gate, W z To reset the gate weight parameters, is the weight parameter of the candidate set, W o for y t With x t and h t-1 The weight parameter [] of the connection matrix multiplication constitutes vector connection, · represents matrix dot multiplication, × represents matrix product, σ represents sigmoid activation function, and φ represents tanh activation function. The mathematical description of σ and φ is as follows:

[0093]

[0094]

[0095] In the exemplary embodiment, the GRU network adopts a 3-layer GRU network structure, and the relevant parameters of the 3-layer GRU network are, for example, iput_size=322, hidden_size=322, num_layers=3, and dropout=0.2. Among them, input_size represents the dimension size of the feature vector input at each moment; hidden_size represents the dimension size of the hidden state in the GRU unit. In the GRU network, the hidden state at each moment is obtained by a series of operations on the input feature vector at the current moment and the hidden state at the previous moment. Hidden_size determines the dimension size of the hidden state, which can also be understood as the dimension size of the feature vector output at each moment of the GRU network; num_layers represents the number of GRU layers in the GRU network, that is, the number of stacked layers of GRU units; dropout is a regularization parameter, which is used to prevent overfitting of the GRU network. In the GRU network, dropout can be applied to the positions of the input feature vector, hidden state, and output feature vector. According to the value of dropout, some input elements are randomly set to 0, and these elements are discarded with a certain probability, thereby reducing the dependency between neurons, reducing the complexity of the model, and improving the generalization ability of the model. By using dropout during the training process, the model can be forced to learn more robust and robust feature representations to reduce overfitting problems.

[0096] Step 303: Input the noisy speech feature into a first linear classifier, and obtain a first feature vector through the first linear classifier.

[0097] In an illustrative embodiment, the first linear classifier can be a convolution with kernel_size=1. The parameters of the first linear classifier are: kernel_size=1, input_size=322, out_size=161. Among them, kernel_size refers to the size of the convolution kernel, input_size represents the dimension of the feature vector input at each moment, which is consistent with the input_size of the GRU network, and out_size is the size of the output feature. The size of the output feature of the first linear classifier is set to be equal to the number of channels of the noisy speech amplitude spectrum (x_noisy_mag), which is to make the number of channels of the amplitude spectrum mask obtained subsequently equal to the number of channels of the noisy speech amplitude spectrum, so that the noisy speech amplitude spectrum and the amplitude spectrum mask can be matched, thereby realizing the multiplication of the noisy speech amplitude spectrum and the amplitude spectrum mask.

[0098] The first feature vector is a vector with the same dimension as the amplitude spectrum obtained by transforming the dimension of the feature extracted by the previous GRU network through the first linear classifier. Only when the dimension of the first feature vector is the same as the dimension of the amplitude spectrum, the amplitude spectrum mask obtained according to the first feature vector can be multiplied with the amplitude spectrum. In addition, the first feature vector is also a feature that the first linear classifier further extracts from the noisy speech feature. Based on this, another purpose of the first linear classifier is to further extract features.

[0099] Step 304: Input the first eigenvector into a normalization module, and obtain an amplitude spectrum mask through the normalization module.

[0100] In an exemplary embodiment, the normalization module performs a normalization operation using a sigmoid function to obtain an amplitude spectrum mask. In an exemplary embodiment, the dimension of the amplitude spectrum mask is dim=(32, 161, 400), which is equal to the dimension of the amplitude spectrum of the noisy speech.

[0101] Step 305: input the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module, and multiply the noisy speech amplitude spectrum and the amplitude spectrum mask by the multiplication module to obtain a masked audio amplitude spectrum.

[0102] Through the multiplication in step 305, the amplitudes of the corresponding positions in the amplitude spectrum of the noisy speech and the amplitude spectrum of the masking audio are changed, thereby achieving the purpose of weakening the amplitude of the noise part.

[0103] Step 306: Input the masked audio amplitude spectrum and the noisy speech phase spectrum to an inverse short-time Fourier transform (ISTFT) module, and perform ISTFT on the masked audio amplitude spectrum and the noisy speech phase spectrum through the ISTFT module to obtain a first noise-reduced speech signal.

[0104] ISTFT is the inverse operation of STFT, and the parameters used by ISTFT are the same as those used by STFT in step 301, for example: nfft=320, hop_length=160, win_length=320. Thus, the dimension of the first denoised speech signal obtained is the same as that of the noisy speech signal, for example, dim=(32, 1, 64000).

[0105] Step 307: Input the noisy speech feature into a second linear classifier, and obtain a second feature vector through the second linear classifier.

[0106] In an illustrative embodiment, the second linear classifier can be a convolution with kernel_size=1. The parameters of the second linear classifier are: kernel_size=1, input_size=322, out_size=128. Regarding the out_size of the first linear classifier and the out_size of the second linear classifier, the out_size of the first linear classifier is 161 because it must be the same dimension as the amplitude spectrum, and the second linear classifier is only for further feature extraction, so it is not necessary to be consistent with the first linear classifier. The out_size of the second linear classifier is set to 128 because the categories of the subsequent noise type classifier are 4 (see subsequent description for details), there are not many categories, and the dimension of the previous feature vector is not required to be large. The reason why the out_size of the second linear classifier is set to 128 is that it is an integer multiple of 2 and is easy to handle.

[0107] Step 308: Input the second feature vector into an average pooling (avg_pooling) module, and perform average pooling on the second feature vector through the average pooling module to obtain a third feature vector.

[0108] Average pooling is a pooling operation method used to reduce the size of the feature map. In the embodiment of the present disclosure, the second feature vector is reduced from the dimension [32, 128, 400] to [32, 128, 1] by average pooling, that is, the time dimension is changed from 400 to 1.

[0109] Step 309: Input the third feature vector into a noise type classifier, and obtain the noise category through the noise type classifier.

[0110] In an exemplary embodiment, the noise type classifier can be a convolution with kernel_size=1. The parameters of the noise type classifier are: kernel_size=1, input_size=128, out_size=4. In an exemplary embodiment, the dimension of the noise category is [32,4], where 32 is the batch size (batch_size), and 4 means that the noise is classified into 4 categories. According to the pre-set categories and through training, the required classification results can be obtained. For example, the 4 categories are divided into: steady-state noise, transient noise, human voice noise, and other noise. Among them, steady-state noise includes wind noise, air conditioning, car tire noise and other noises, transient noise includes door slamming, train whistle, thunder and lightning and other noises, human voice noise includes background human voices, footsteps and other noises in cafes, shopping malls, etc., and other noises are noises that cannot be classified into steady-state noise, transient noise and human voice noise.

[0111] Step 103: Obtain a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal.

[0112] Figure 6 FIG. 1 is a schematic flow chart of steps for obtaining a noise intensity estimation value according to an exemplary embodiment. Figure 6 As shown, in the exemplary embodiment, step 103 includes the following steps 601 to 603 .

[0113] Step 601: Obtain an estimated value of a noise signal in the noisy speech signal according to a first noise-reduced speech signal and a noisy speech signal.

[0114] In the exemplary embodiment, the noise signal estimation value in the noisy speech signal is obtained by subtracting the noisy speech signal from the first noise reduction speech signal. Since the noise signal is a part of the noisy speech signal, the noise signal obtained by subtracting the noisy speech signal from the first noise reduction speech signal is not necessarily an accurate noise signal, but only an estimation value, so it is referred to as the noise signal estimation value here. Specifically, the noise signal estimation value in the noisy speech signal is obtained by the following formula:

[0115] noise 2 =x_noise-x_denoise

[0116] Among them, x_noise is the noisy speech signal, x_denoise is the first noise reduction speech signal, noise 2 is the estimated value of the noise signal in the noisy speech signal.

[0117] Step 602: Obtain the noise signal power and the first noise reduction speech signal power according to the noise signal estimation value and the first noise reduction speech signal respectively.

[0118] In an exemplary embodiment, the noise signal power and the first noise reduction speech signal power are obtained by the following formula:

[0119]

[0120]

[0121] Among them, i is the signal sampling point, N is the total number of sampling points, noise_spe 2 is the noise signal power, noise 2 [i] is the signal amplitude of the i-th signal sampling point of the noise signal estimation value in the noisy speech signal, clean_spe 2 is the power of the first denoised speech signal, and x_denoise[i] is the signal amplitude of the i-th signal sampling point of the first denoised speech signal.

[0122] Step 603: Obtain a noise intensity estimation value according to the noise signal power and the first noise reduction speech signal power.

[0123] In an exemplary embodiment, step 603 may specifically include: determining a signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power as a noise intensity estimation value.

[0124] In an exemplary embodiment, the signal-to-noise ratio, i.e., the noise intensity estimation value, is obtained by the following formula:

[0125]

[0126] Among them, snr is the signal-to-noise ratio.

[0127] Step 104: Mix the first noise-reduced speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise-reduced speech signal.

[0128] For transient noises such as door slamming, whistle honking, thunder and lightning, the noise has a short duration and has little impact on the voice, and the short-term transient noise has little negative impact on the listener's voice hearing perception. At the same time, compared with the noisy voice signal containing long-lasting steady-state noise, the first noise-reduced voice signal obtained after the noise reduction of the noisy voice signal containing transient noise has a better voice hearing perception. In this case, if the first noise-reduced voice signal obtained after the noise reduction of the noisy voice signal containing transient noise is mixed with the noisy voice signal again, the transient noise in the noisy voice signal will not enhance the clarity or hearing perception of the voice part, but the transient noise will make the hearing perception of the mixed voice signal worse.

[0129] In contrast, for steady-state noises such as long-lasting wind noise, air conditioning noise, car engine noise, tire noise, etc., since the noise lasts for a long time and even reaches or exceeds the speech duration, it has a greater impact on the induced voice, and the continuous steady-state noise has a great negative impact on the listener's speech auditory perception. At the same time, compared with the noisy speech signal containing transient noise, the first noise-reduced speech signal obtained after the noise-reduced speech signal containing steady-state noise will cause poor speech hearing due to excessive noise filtering. In this case, if the first noise-reduced speech signal obtained after the noise-reduced speech signal containing steady-state noise is mixed with the noisy speech signal to a certain extent, although a certain amplification will occur in the noise part, the speech part will also be amplified at the same time. Due to the superposition of the speech in the noisy speech signal and the first noise-reduced speech signal, the hearing performance of the speech part in the mixed signal will be further enhanced. Therefore, on the whole, the first noise-reduced speech signal obtained after the noise-reduced speech signal containing steady-state noise is mixed with the noisy speech signal to a certain extent, and the second noise-reduced speech signal can be obtained.

[0130] The listening experience of speech signals is mainly concentrated in three aspects: first, the speech quality in the signal; second, the background noise quality in the signal; and third, the overall audio quality. If the speech quality in the signal is high, the listening experience will be improved. However, excessive filtering of noise by noise reduction processing will reduce the speech quality and also reduce the overall audio quality. Although noise reduction processing can remove background noise and improve the background noise quality, the decline in speech quality and overall audio quality will lead to a decline in listening experience.

[0131] When the noise in the noisy speech signal is steady-state noise, the first noise reduction speech signal and the noisy speech signal are mixed. Although the noise removal quality of the background noise in the signal will be reduced, the speech quality in the signal will be enhanced, and the overall audio quality will also be improved. Therefore, the overall listening experience of the speech signal can still be improved.

[0132] Figure 7 FIG. 1 is a flow chart showing the steps of mixing a first noise reduction speech signal and a noisy speech signal according to an exemplary embodiment. Figure 7 As shown, in the exemplary embodiment, step 104 includes the following steps 701 to 7032 .

[0133] Step 701: Determine whether it is necessary to mix the first noise reduction speech signal and the noisy speech signal according to the noise category.

[0134] In an exemplary embodiment, the noise category includes at least steady-state noise. In an exemplary embodiment, the noise category may also include transient noise and other categories. Step 701 specifically includes the following two cases:

[0135] When the noise category is steady-state noise, determining that the first noise reduction speech signal and the noisy speech signal need to be mixed; or

[0136] When the noise type is not steady-state noise, it is determined that there is no need to perform signal mixing on the first noise reduction speech signal and the noisy speech signal.

[0137] Step 702: When there is no need to mix the first noise reduction speech signal and the noisy speech signal, do not mix the first noise reduction speech signal and the noisy speech signal.

[0138] Step 703: When it is necessary to mix the first noise reduction speech signal and the noisy speech signal, perform the following steps:

[0139] Step 7031, obtaining a noise mixing coefficient according to the noise intensity estimation value;

[0140] Step 7032: Mix the first noise-reduced speech signal and the noisy speech signal according to the noise mixing coefficient to obtain a second noise-reduced speech signal.

[0141] In an exemplary embodiment, step 7031 includes:

[0142] The noise coefficient is obtained according to the following formula:

[0143]

[0144] Wherein, β is the noise intensity estimation value, A is the preset minimum value of the mixed noise coefficient, B is the preset maximum value of the mixed noise coefficient, m is the preset lower limit value of the noise intensity screening range, and n is the preset upper limit value of the noise intensity screening range.

[0145] Figure 8 is a schematic diagram of an embodiment of a mapping relationship between a noise mixing coefficient and a noise intensity estimation value according to an exemplary embodiment, such as Figure 8 As shown, the horizontal axis of the coordinate system is the noise intensity estimation value, wherein the noise intensity estimation value is the signal-to-noise ratio, and the vertical axis of the coordinate system is the mixing noise coefficient. Figure 8 In the illustrated embodiment, the value interval of the noise intensity estimation value is set to [-5db, 15db]. Then, when the noise intensity estimation value is less than -5db, the noise mixing coefficient is set to 0.01. When the noise intensity estimation value is greater than 15db, the noise mixing coefficient is set to 0.05. In the interval of [-5db, 15db], the noise mixing coefficient and the noise intensity estimation value are in a linear function relationship, that is:

[0146] α=0.002*β+0.02

[0147] In this way, Figure 8 In the exemplary embodiment shown, the variation interval of the noise mixing coefficient is [0.01, 0.05]. It should be noted that the variation interval of the noise mixing coefficient is [0.01, 0.05] for illustrative purposes only. In practical applications, the variation interval of the noise mixing coefficient can be appropriately adjusted as needed, and the interval [-5db, 15db] in which the noise mixing coefficient is adjusted with the noise intensity estimation value can also be appropriately adjusted as needed.

[0148] In an exemplary embodiment, step 7032 includes:

[0149] The second noise reduction speech signal is obtained by the following formula:

[0150] x_blend=(1-α)*x_denoise+α*x_noise

[0151] Among them, x_blend is the second denoised speech signal, α is the mixing noise coefficient, x_denoise is the first denoised speech signal, and x_noise is the noisy speech signal.

[0152] Thus, it can be seen that when the signal-to-noise ratio is low, the noise mixing coefficient is small, so that the second noise reduction speech signal contains more first noise reduction speech signal components and less original noisy speech signal components. When the signal-to-noise ratio is high, the noise mixing coefficient is large, and the second noise reduction speech signal contains more original noisy speech signal components and less first noise reduction speech signal components.

[0153] A low signal-to-noise ratio means that the noise is relatively large, so that the first noise reduction voice signal component contained in the second noise reduction voice signal with more components can reduce the impact of noise, and the original noisy voice signal component contained in the second noise reduction voice signal with less components can make up for the clean voice part with a higher degree of filtering in the first noise reduction voice signal, and improve the performance of the clean voice part, so that the second noise reduction voice signal has improved listening experience compared with the first noise reduction voice signal. A high signal-to-noise ratio means that the noise is relatively small, so that the first noise reduction voice signal component contained in the second noise reduction voice signal with less components can enhance the clean voice part in the original noisy voice signal, and the noise in the original noisy voice signal component contained in the second noise reduction voice signal with more components is relatively small, and the enhanced clean voice part has a weakening effect on the noise part in listening experience, so that the second noise reduction voice signal has improved listening experience compared with the first noise reduction voice signal. Therefore, the overall listening experience of the second noise reduction voice signal is better than that of the first noise reduction voice signal before mixing.

[0154] Since the GRU network, linear classifier, normalization module, average pooling module and noise type classifier are used in the embodiments of the present disclosure, these structures belong to artificial intelligence neural networks and need to be trained to achieve the required functions and obtain the desired results. Therefore, the embodiments of the present disclosure also include a training process. Fig. 9 FIG. 1 is a flow chart showing the steps of training a noise reduction classification network model according to an exemplary embodiment. Fig. 9 As shown, in the exemplary embodiment, the speech noise reduction processing method of the embodiment of the present disclosure also includes the following steps 901 to 903.

[0155] Step 901: Mix a clean speech sample signal and a noise sample signal to obtain a noisy speech sample signal.

[0156] In an exemplary embodiment, the noisy speech sample signal is obtained by mixing a clean speech sample signal and a noise sample signal, and the formula is as follows:

[0157]

[0158]

[0159]

[0160] noisy_signal=clean+scalar*noise 1

[0161] Among them, i is the signal sampling point, N is the total number of sampling points, noise_spe 1 Represents the power of the noise sample signal, noise 1 [i] is the signal amplitude of the i-th signal sampling point of the noise sample signal, clean_spe 1 represents the power of the clean speech sample signal, clean[i] represents the signal amplitude of the i-th signal sampling point of the clean speech sample signal, snr 1 is a preset signal-to-noise ratio. In the exemplary embodiment, snr 1 The value range is [-5db, 15db], scalar is the scaling factor of the noise signal, noisy_signal is the speech signal after mixing, which is used as the input of the deep noise reduction network, clean is the clean speech sample signal, and noise is the noise sample signal.

[0162] In an exemplary embodiment, according to the four categories of steady-state noise, transient noise, human voice noise, and other noises set in advance, when making training sample data, the noise type labels 0, 1, 2, and 3 are obtained according to the selected noise, where 0 represents steady-state noise, 1 represents transient noise, 2 represents human voice noise, and 3 represents other noise. In an exemplary embodiment, the sampling rate of the clean speech sample signal is 16000 Hz, and 4s of data are randomly intercepted during training as training. If the duration of the clean speech sample signal is less than 4s (seconds), zeros to 4s are randomly filled on the left and right of the clean speech sample signal. In the above formula for mixing the clean speech sample signal and the noise sample signal to obtain the noisy speech sample signal, N=64000 is obtained according to the sampling rate of 16000 Hz and the duration of the 4s clean speech sample signal. In an exemplary embodiment, the length of the noise sample signal is consistent with the length of the clean speech sample signal.

[0163] Step 902: Input the noisy speech sample signal into a noise reduction classification network model composed of an STFT module, a GRU network, a first linear classifier, a normalization module, a multiplication module, an ISTFT module, a second linear classifier, an average pooling module and a noise type classifier to obtain a noise classification prediction result and a noise reduction speech prediction signal.

[0164] Step 903: According to the difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the clean speech sample signal, the noise reduction classification network model is trained to adjust the parameters in the noise reduction classification network model.

[0165] Fig.10 FIG. 1 is a flow chart of steps for adjusting parameters in a noise reduction classification network model according to differences during training according to an exemplary embodiment. Fig.10 As shown, step 903 specifically includes the following steps 1001 to 1004.

[0166] Step 1001: Establish a cross entropy loss function based on the noise classification prediction result and the preset noise classification label.

[0167] The cross entropy loss function, namely CE Loss (Cross Entropy Loss), has the following formula:

[0168]

[0169] Among them, ce_loss is the cross entropy loss, B is the number of batch samples, i is the sample number, M is the number of categories, c is the category number, y is the ic Is the i-th sample of category c, y ic The value is 0 or 1, p ic is the probability of the ith sample being identified as category c. In the disclosed embodiment, the cross entropy loss function characterizes the difference between the noise classification prediction result and the preset noise classification label.

[0170] Step 1002: Establish a scale-invariant signal-to-noise ratio loss function based on the noise-reduced speech prediction signal and the clean speech sample signal.

[0171] The scale-invariant signal-to-noise ratio loss function, namely SI-SNR Loss (Scale Invariant Signal to Noise Ratio), is formulated as follows:

[0172]

[0173] in,

[0174]

[0175]

[0176] Among them, sisnr_loss is the scale-invariant signal-to-noise ratio loss, s target Represents the predicted speech signal The projection signal on the label speech signal s, s represents the label speech signal, is the speech signal predicted by the model, represents the sum of the products of the corresponding positions of the predicted speech signal and the labeled speech signal, ||s|| 2 Represents the power of the label speech signal, and the calculation formula is as follows:

[0177]

[0178]

[0179] Where, N is the number of sampling points.

[0180] Similarly, ||s target || 2 Indicates target The power of the signal, ||e noise || 2 Indicates e noise The power of the signal is calculated in the same way.

[0181] In the disclosed embodiment, the scale-invariant signal-to-noise ratio loss function characterizes the difference between the noise-reduced speech prediction signal and the clean speech sample signal.

[0182] Step 1003: According to the cross entropy loss function and the scale-invariant signal-to-noise ratio loss function, an overall loss function of the denoising classification network model is obtained.

[0183] In an exemplary embodiment, the overall loss function is as follows:

[0184] total_loss = α 1 *ce_loss+α 2 *sisnr_loss

[0185] Among them, α 1 is the weight of ce_loss, α 2 is the weight of sisnr_loss. In the exemplary embodiment, α 1 and α 2 Both are set to 0.5.

[0186] Step 1004: According to the overall loss function, the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the denoising classification network model are adjusted through back propagation until the loss value of the overall loss function converges to the expected value or reaches the number of training iterations.

[0187] In the exemplary embodiment, although the denoising classification network model includes an STFT module and an ISTFT module, STFT and ISTFT still adopt a traditional method, and the parameters used are fixed, so there is no need to train and adjust the parameters.

[0188] The speech noise reduction processing method of the embodiment of the present disclosure performs deep noise reduction processing and classification on the noisy speech signal, and adaptively mixes the speech signal after noise reduction with the original noisy speech signal according to the noise intensity estimation value and the noise category, thereby alleviating the problem of decreased hearing caused by excessive noise filtering. Although the speech signal after noise reduction is mixed with the original noisy speech signal to enhance the performance of the noise part, the speech quality and the overall audio quality are also improved at the same time, thus improving the hearing of the speech signal as a whole. At the same time, in the speech noise reduction processing method of the embodiment of the present disclosure, the noise reduction and noise type prediction are simultaneously completed by using the artificial intelligence noise reduction classification network model, which enhances the correlation between the noise reduction structure and the prediction result, and helps to improve the accuracy of the noise type prediction result in judging whether to mix the signal. Moreover, in the speech noise reduction processing method of the embodiment of the present disclosure, the mixing noise coefficient varies according to the size of the noise intensity estimation value. When the noise intensity estimation value is low, the second noise reduction speech signal retains more first noise reduction speech signal components and fewer original noisy speech signal components. Because the noise component in the original noisy speech signal is high when the noise intensity estimation value is low, the noise reduction process has a greater impact on the pure speech part, resulting in a higher degree of filtering of the pure speech part in the first noise reduction speech signal. Therefore, the second noise reduction speech signal retains fewer first noise reduction speech signal components, and more original noisy speech signal components are used to compensate for the pure speech part with a higher degree of filtering in the first noise reduction speech signal. Although a certain amount of noise components are introduced, the compensation for the pure speech part with a higher degree of filtering improves the performance of the speech part from both the pure speech perspective and the overall perspective of the speech signal, thereby helping to improve the noise reduction speech signal. The second denoised speech signal has a better hearing experience than the first denoised speech signal; when the noise intensity estimation value is high, the second denoised speech signal retains fewer first denoised speech signal components and more original noisy speech signal components. Because the noise component in the original noisy speech signal is low when the noise intensity estimation value is high, the noise reduction process has less impact on the pure speech part, so that the degree of filtering of the pure speech part in the first denoised speech signal is low, so the second denoised speech signal retains more first denoised speech signal components. Only with fewer original noisy speech signal components, the performance of the pure speech part in the first denoised speech signal and the overall performance of the denoised speech signal can be improved, thereby helping to improve the hearing experience of the denoised speech signal, so that the hearing experience of the second denoised speech signal is better than the first denoised speech signal.

[0189] Fig.11 FIG. 1 is a schematic flow chart of a specific application scenario of a speech noise reduction processing method according to an exemplary embodiment. Fig.11 As shown, the process includes the following steps 1101 to 1114.

[0190] Step 1101: construct a denoising classification network model to be trained, and then execute step 1102.

[0191] The denoising classification network model includes an STFT module, a GRU network, a first linear classifier, a normalization module, a multiplication module, an ISTFT module, a second linear classifier, an average pooling module, and a noise type classifier. The structure of the model can be found in Figure 4 The topological diagram of the process of deep noise reduction processing and classification of noisy speech signals is shown. Among them, the STFT module is used to perform STFT on the input signal to obtain the amplitude spectrum and phase spectrum; the GRU network is used to extract the features of the amplitude spectrum to obtain the audio features; the first linear classifier is used to process the audio features to obtain the first feature vector; the normalization module is used to perform a normalization operation on the first feature vector using the sigmoid function to obtain the amplitude spectrum mask; the multiplication module is used to multiply the amplitude spectrum and the amplitude spectrum mask to obtain the masked amplitude spectrum; the ISTFT module is used to perform ISTFT on the masked amplitude spectrum and phase spectrum to obtain the noise reduction signal; the second linear classifier is used to process the audio features to obtain the second feature vector; the average pooling module is used to average pool the second feature vector to obtain the third feature vector with a time dimension of 1; the noise type classifier is used to obtain the noise category result according to the third feature vector. In the noise reduction classification network model, the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier belong to the artificial intelligence neural network structure, and its weights and other parameters are optimized through training.

[0192] Step 1102 , obtaining a clean speech sample signal and a noise sample signal, and mixing the clean speech sample signal and the noise sample signal to obtain a noisy speech sample signal, and then executing step 1103 .

[0193] In an exemplary embodiment, the duration of a clean voice sample signal is 4s. If the duration of a voice signal for producing a clean voice sample signal exceeds 4s, 4s of the signal are randomly intercepted to generate a clean voice sample signal. If the duration of the voice signal is less than 4s, zeros are filled before and after the voice signal to generate a clean voice sample signal. The duration of the noisy voice sample signal is consistent with that of the clean voice sample signal, which is also 4s. The noisy voice sample signal is a noise sample signal mixed with a 4s clean voice sample signal. In an exemplary embodiment, the noise sample signal can be a steady-state noise sample, a transient noise sample, a human voice noise sample, or other noise samples.

[0194] Step 1103: input the noisy speech sample signal into the noise reduction classification network model to be trained to obtain the noise classification prediction result and the noise reduction speech prediction signal, and then execute step 1104.

[0195] Step 1104: establish a cross entropy loss function based on the noise classification prediction result and the preset noise classification label, establish a scale-invariant signal-to-noise ratio loss function based on the noise reduction speech prediction signal and the clean speech sample signal, and obtain the overall loss function of the noise reduction classification network model based on the cross entropy loss function and the scale-invariant signal-to-noise ratio loss function, and then execute step 1105.

[0196] Step 1105: According to the overall loss function, adjust the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the denoising classification network model through back propagation, and then execute step 1106.

[0197] Step 1106: Determine whether the loss value of the overall loss function converges to the expected value. If so, execute step 1108; otherwise, execute step 1107.

[0198] Step 1107 , determine whether the number of training iterations has been reached, if so, execute step 1108 , otherwise execute step 1102 .

[0199] Step 1108 , obtaining a noisy speech signal, inputting the noisy speech signal into the trained noise reduction classification network model, obtaining a first noise reduction speech signal and a noise category, and then executing step 1109 .

[0200] Step 1109: Obtain an estimated value of a noise signal in the noisy speech signal according to the first noise reduction speech signal and the noisy speech signal, and then execute step 1110.

[0201] Step 1110: Obtain the noise signal power and the first noise reduction speech signal power according to the noise signal estimation value and the first noise reduction speech signal respectively, and then execute step 1111.

[0202] Step 1111: Obtain a signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power according to the noise signal power and the first noise reduction speech signal power, and then execute step 1112.

[0203] In an exemplary embodiment, the signal-to-noise ratio, i.e., the noise intensity estimation value, is obtained by the following formula:

[0204]

[0205] Where, snr is the signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power, clean_spe 2 is the first noise reduction speech signal power, noise_spe 2 is the noise signal power.

[0206] Step 1112: determine whether the noise type is steady-state noise, if yes, execute 1113, otherwise, end the process.

[0207] Step 1113: Obtain a noise mixing coefficient according to a signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power, and then execute step 1114.

[0208] In an exemplary embodiment, the noise coefficient is obtained according to the following formula:

[0209]

[0210] Among them, β is the signal-to-noise ratio (noise intensity estimation) between the noise signal power and the first noise reduction speech signal power, that is, snr in step 1111, A is the preset minimum value of the mixing noise coefficient, B is the preset maximum value of the mixing noise coefficient, m is the preset lower limit value of the noise intensity screening range, and n is the preset upper limit value of the noise intensity screening range.

[0211] Step 1114: Mix the first noise-reduced speech signal and the noisy speech signal according to the noise mixing coefficient to obtain a second noise-reduced speech signal.

[0212] In an exemplary embodiment, the second noise reduction speech signal is obtained by the following formula:

[0213] x_blend=(1-α)*x_denoise+α*x_noise

[0214] Among them, x_blend is the second denoised speech signal, α is the mixing noise coefficient, x_denoise is the first denoised speech signal, and x_noise is the noisy speech signal.

[0215] At this point, the application scenario process is completed.

[0216] Fig.12is a schematic diagram of the logical structure of a speech noise reduction processing device according to an exemplary embodiment. Fig.12 As shown, the speech noise reduction processing device includes a signal acquisition module 1201, a noise reduction classification module 1202, a noise intensity acquisition module 1203 and a noise reduction speech acquisition module 1204.

[0217] The signal acquisition module 1201 is configured to acquire a noisy speech signal;

[0218] The noise reduction classification module 1202 is configured to perform deep noise reduction processing and classification on the noisy speech signal to obtain a first noise reduction speech signal and a noise category;

[0219] The noise intensity obtaining module 1203 is configured to obtain a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal;

[0220] The noise reduction speech acquisition module 1204 is configured to perform signal mixing on the first noise reduction speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise reduction speech signal.

[0221] In an exemplary embodiment, the noise reduction classification module 1202 includes:

[0222] The STFT unit is configured to input the noisy speech signal into a short-time Fourier transform (STFT) module, and perform STFT on the noisy speech signal through the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum;

[0223] The feature extraction unit is configured to input the noisy speech amplitude spectrum into the GRU network, extract features of the noisy speech amplitude spectrum through the GRU network, and obtain the noisy speech features;

[0224] A first feature acquisition unit is configured to input the noisy speech feature into a first linear classifier, and obtain a first feature vector through the first linear classifier;

[0225] A mask acquisition unit is configured to input the first feature vector into a normalization module, and obtain an amplitude spectrum mask through the normalization module;

[0226] The masked audio amplitude spectrum acquisition unit is configured to input the noisy speech amplitude spectrum and the amplitude spectrum mask into the multiplication module, and multiply the noisy speech amplitude spectrum and the amplitude spectrum mask by the multiplication module to obtain the masked audio amplitude spectrum;

[0227] The ISTFT unit is configured to input the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module, and perform ISTFT on the masked audio amplitude spectrum and the noisy speech phase spectrum input by the ISTFT module to obtain a first noise reduction speech signal;

[0228] A second feature acquisition unit is configured to input the noisy speech feature into a second linear classifier, and obtain a second feature vector through the second linear classifier;

[0229] A pooling unit is configured to input the second feature vector into an average pooling (avg_pooling) module, and perform average pooling on the second feature vector through the average pooling module to obtain a third feature vector;

[0230] The classification unit is configured to input the third feature vector into a noise type classifier, and obtain a noise category through the noise type classifier.

[0231] In an exemplary embodiment, the speech noise reduction processing device further includes:

[0232] A sample mixing module is configured to mix a clean speech sample signal and a noise sample signal to obtain a noisy speech sample signal;

[0233] The prediction result acquisition module is configured to execute the input of the noisy speech sample signal into the noise reduction classification network model composed of the STFT module, the GRU network, the first linear classifier, the normalization module, the multiplication module, the ISTFT module, the second linear classifier, the average pooling module and the noise type classifier to obtain the noise classification prediction result and the noise reduction speech prediction signal;

[0234] The model training module is configured to train the noise reduction classification network model based on the difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the clean speech sample signal, so as to adjust the parameters in the noise reduction classification network model.

[0235] In an exemplary embodiment, the model training module includes:

[0236] A first loss function establishment submodule is configured to establish a cross entropy loss function according to the noise classification prediction result and the preset noise classification label;

[0237] A second loss function establishment submodule is configured to establish a scale-invariant signal-to-noise ratio loss function based on the noise-reduced speech prediction signal and the clean speech sample signal;

[0238] The overall loss function obtaining submodule is configured to obtain an overall loss function of the denoising classification network model according to the cross entropy loss function and the scale-invariant signal-to-noise ratio loss function;

[0239] The model parameter adjustment submodule is configured to adjust the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the denoising classification network model according to the overall loss function through back propagation until the loss value of the overall loss function converges to the expected value or reaches the number of training iterations.

[0240] In an exemplary embodiment, the noise intensity obtaining module 1203 includes:

[0241] A noise signal acquisition submodule is configured to obtain a noise signal estimation value in the noisy speech signal according to the first noise reduction speech signal and the noisy speech signal;

[0242] A signal power acquisition submodule is configured to obtain the noise signal power and the first noise reduction speech signal power according to the noise signal estimation value and the first noise reduction speech signal respectively;

[0243] The noise intensity obtaining submodule is configured to obtain a noise intensity estimation value according to the noise signal power and the first noise reduction speech signal power.

[0244] The noise intensity obtaining submodule is further configured to execute: determining a signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power as a noise intensity estimation value.

[0245] In an exemplary embodiment, the noise reduction speech acquisition module 1204 is further configured to execute:

[0246] Determining whether it is necessary to perform signal mixing on the first noise reduction speech signal and the noisy speech signal according to the noise category;

[0247] When there is no need to mix the first noise reduction speech signal and the noisy speech signal, not mixing the first noise reduction speech signal and the noisy speech signal;

[0248] When the first noise reduction speech signal and the noisy speech signal need to be mixed, the following steps are performed:

[0249] According to the noise intensity estimation value, the mixed noise coefficient is obtained;

[0250] The first noise reduction speech signal and the noisy speech signal are mixed according to the noise mixing coefficient to obtain a second noise reduction speech signal.

[0251] In the exemplary embodiment, the noise category includes steady-state noise and transient noise; the noise reduction speech acquisition module 1204 is further configured to execute:

[0252] When the noise category is steady-state noise, determining that the first noise reduction speech signal and the noisy speech signal need to be mixed; or

[0253] When the noise type is not steady-state noise, it is determined that there is no need to perform signal mixing on the first noise reduction speech signal and the noisy speech signal.

[0254] In an exemplary embodiment, the noise mixing coefficient is obtained according to the noise intensity estimation value, including:

[0255] The noise coefficient is obtained according to the following formula:

[0256]

[0257] Wherein, β is the noise intensity estimation value, A is the preset minimum value of the mixed noise coefficient, B is the preset maximum value of the mixed noise coefficient, m is the preset lower limit value of the noise intensity screening range, and n is the preset upper limit value of the noise intensity screening range.

[0258] In an exemplary embodiment, mixing a first noise reduction speech signal and a noisy speech signal according to a noise mixing coefficient to obtain a second noise reduction speech signal includes:

[0259] The second noise reduction speech signal is obtained by the following formula:

[0260] x_blend=(1-α)*x_denoise+α*x_noise

[0261] Among them, x_blend is the second denoised speech signal, α is the mixing noise coefficient, x_denoise is the first denoised speech signal, and x_noise is the noisy speech signal.

[0262] The speech noise reduction processing device of the embodiment of the present disclosure performs deep noise reduction processing and classification on the noisy speech signal, and adaptively mixes the noise-reduced speech signal with the original noisy speech signal according to the noise intensity estimation value and the noise category, thereby alleviating the problem of decreased hearing caused by excessive noise filtering. Although the noise-reduced speech signal is mixed with the original noisy speech signal to enhance the performance of the noise part, the speech quality and the overall audio quality are also improved at the same time, thereby improving the hearing of the speech signal as a whole. At the same time, in the speech noise reduction processing device of the embodiment of the present disclosure, the noise reduction and noise type prediction are simultaneously completed by using the artificial intelligence noise reduction classification network model, which enhances the correlation between the noise reduction structure and the prediction result, and helps to improve the accuracy of the noise type prediction result in judging whether to mix the signal. Moreover, in the speech noise reduction processing device of the embodiment of the present disclosure, the mixing noise coefficient varies according to the size of the noise intensity estimation value. When the noise intensity estimation value is low, the second noise reduction speech signal retains more first noise reduction speech signal components and fewer original noisy speech signal components. Because the noise component in the original noisy speech signal is high when the noise intensity estimation value is low, the noise reduction process has a greater impact on the pure speech part, resulting in a higher degree of filtering of the pure speech part in the first noise reduction speech signal. Therefore, the second noise reduction speech signal retains fewer first noise reduction speech signal components, and more original noisy speech signal components are used to compensate for the pure speech part with a higher degree of filtering in the first noise reduction speech signal. Although a certain amount of noise components are introduced, the compensation for the pure speech part with a higher degree of filtering improves the performance of the speech part from both the pure speech perspective and the overall perspective of the speech signal, thereby helping to improve the noise reduction speech signal. The second denoised speech signal has a better hearing experience than the first denoised speech signal; when the noise intensity estimation value is high, the second denoised speech signal retains fewer first denoised speech signal components and more original noisy speech signal components. Because the noise component in the original noisy speech signal is low when the noise intensity estimation value is high, the noise reduction process has less impact on the pure speech part, so that the degree of filtering of the pure speech part in the first denoised speech signal is low, so the second denoised speech signal retains more first denoised speech signal components. Only with fewer original noisy speech signal components, the performance of the pure speech part in the first denoised speech signal and the overall performance of the denoised speech signal can be improved, thereby helping to improve the hearing experience of the denoised speech signal, so that the hearing experience of the second denoised speech signal is better than the first denoised speech signal.

[0263] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0264] Regarding the speech noise reduction processing device in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the speech noise reduction processing method, and will not be elaborated here.

[0265] It should be noted that: the above embodiment only illustrates the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0266] Fig.13 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. In some embodiments, the electronic device is a server. The electronic device 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein at least one program code is stored in the memory 1302, and the at least one program code is loaded and executed by the processor 1301 to implement the speech noise reduction processing method provided in each of the above embodiments. Of course, the electronic device 1300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 1300 may also include other components for implementing device functions, which will not be repeated here.

[0267] The above are only preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A speech noise reduction processing method, comprising: Acquire a noisy speech signal; Performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise-reduced speech signal and a noise category; Obtaining a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal; According to the noise intensity estimation value and the noise category, the first noise-reduced speech signal and the noisy speech signal are mixed to obtain a second noise-reduced speech signal.

2. The method for speech noise reduction according to claim 1, characterized in that: The performing deep noise reduction processing and classification on the noisy speech signal to obtain a first noise reduction speech signal and a noise category includes: Inputting the noisy speech signal into a short-time Fourier transform (STFT) module, and performing STFT on the noisy speech signal by the STFT module to obtain a noisy speech amplitude spectrum and a noisy speech phase spectrum; Inputting the noisy speech amplitude spectrum into a GRU network, and performing feature extraction on the noisy speech amplitude spectrum through the GRU network to obtain noisy speech features; Inputting the noisy speech feature into a first linear classifier, and obtaining a first feature vector through the first linear classifier; Inputting the first feature vector into a normalization module, and obtaining an amplitude spectrum mask through the normalization module; Inputting the noisy speech amplitude spectrum and the amplitude spectrum mask into a multiplication module, and multiplying the noisy speech amplitude spectrum and the amplitude spectrum mask by the multiplication module to obtain a masked audio amplitude spectrum; Input the masked audio amplitude spectrum and the noisy speech phase spectrum into an inverse short-time Fourier transform (ISTFT) module, and perform ISTFT on the masked audio amplitude spectrum and the noisy speech phase spectrum input by the ISTFT module to obtain the first noise reduction speech signal; Inputting the noisy speech feature into a second linear classifier, and obtaining a second feature vector through the second linear classifier; Inputting the second feature vector into an average pooling module, and performing average pooling on the second feature vector through the average pooling module to obtain a third feature vector; The third feature vector is input into a noise type classifier, and the noise category is obtained through the noise type classifier.

3. The method for speech noise reduction according to claim 2, characterized in that: The speech noise reduction processing method also includes: The clean speech sample signal and the noise sample signal are mixed to obtain a noisy speech sample signal; Inputting the noisy speech sample signal into a noise reduction classification network model composed of the STFT module, the GRU network, the first linear classifier, the normalization module, the multiplication module, the ISTFT module, the second linear classifier, the average pooling module and the noise type classifier to obtain a noise classification prediction result and a noise reduction speech prediction signal; According to the difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the clean speech sample signal, the noise reduction classification network model is trained to adjust the parameters in the noise reduction classification network model.

4. The method for speech noise reduction according to claim 3, characterized in that: The training of the noise reduction classification network model according to the difference between the noise classification prediction result and the preset noise classification label, and the difference between the noise reduction speech prediction signal and the clean speech sample signal to adjust the parameters in the noise reduction classification network model includes: Establishing a cross entropy loss function according to the noise classification prediction result and the preset noise classification label; Establishing a scale-invariant signal-to-noise ratio loss function according to the noise-reduced speech prediction signal and the clean speech sample signal; According to the cross entropy loss function and the scale-invariant signal-to-noise ratio loss function, an overall loss function of the denoising classification network model is obtained; According to the overall loss function, the parameters of the GRU network, the first linear classifier, the normalization module, the second linear classifier, the average pooling module and the noise type classifier in the denoising classification network model are adjusted through back propagation until the loss value of the overall loss function converges to the expected value or reaches the number of training iterations.

5. The method for speech noise reduction according to claim 1, characterized in that: The step of obtaining a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal includes: Obtaining a noise signal estimation value in the noisy speech signal according to the first noise reduction speech signal and the noisy speech signal; Obtaining noise signal power and first noise reduction speech signal power according to the noise signal estimation value and the first noise reduction speech signal respectively; The noise intensity estimation value is obtained according to the noise signal power and the first noise reduction speech signal power.

6. The method for speech noise reduction according to claim 5, characterized in that: The obtaining the noise intensity estimation value according to the noise signal power and the first noise reduction speech signal power includes: The signal-to-noise ratio between the noise signal power and the first noise reduction speech signal power is determined as the noise intensity estimation value.

7. The method for speech noise reduction according to claim 1, characterized in that: The step of mixing the first noise reduction speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise reduction speech signal includes: Determining whether it is necessary to perform signal mixing on the first noise reduction speech signal and the noisy speech signal according to the noise category; When there is no need to mix the first noise reduction speech signal and the noisy speech signal, do not mix the first noise reduction speech signal and the noisy speech signal; or In the case where the first noise reduction speech signal and the noisy speech signal need to be mixed, the following steps are performed: Obtaining a noise mixing coefficient according to the noise intensity estimation value; The first noise-reduced speech signal and the noisy speech signal are mixed according to the noise mixing coefficient to obtain the second noise-reduced speech signal.

8. The method for speech noise reduction according to claim 7, characterized in that: The noise category includes at least steady-state noise; The determining, according to the noise category, whether it is necessary to perform signal mixing on the first noise reduction speech signal and the noisy speech signal comprises: When the noise category is the steady-state noise, determining that the first noise reduction speech signal and the noisy speech signal need to be mixed; or When the noise category is not the steady-state noise, it is determined that there is no need to perform signal mixing on the first noise reduction speech signal and the noisy speech signal.

9. The method for speech noise reduction according to claim 7, characterized in that: The step of obtaining a noise mixing coefficient according to the noise intensity estimation value includes: The noise mixing coefficient is obtained according to the following formula: Wherein, β is the noise intensity estimation value, A is the preset minimum value of the mixed noise coefficient, B is the preset maximum value of the mixed noise coefficient, m is the preset lower limit value of the noise intensity screening range, and n is the preset upper limit value of the noise intensity screening range.

10. The method for speech noise reduction according to claim 7, characterized in that: The step of mixing the first noise reduction speech signal and the noisy speech signal according to the noise mixing coefficient to obtain the second noise reduction speech signal includes: The second noise reduction speech signal is obtained by the following formula: x_blend=(1-α)*x_denoise+α*x_noise Among them, x_blend is the second denoised speech signal, α is the mixing noise coefficient, x_denoise is the first denoised speech signal, and x_noise is the noisy speech signal.

11. A speech noise reduction processing device, characterized in that: include: A signal acquisition module is configured to acquire a noisy speech signal; A noise reduction classification module is configured to perform deep noise reduction processing and classification on the noisy speech signal to obtain a first noise reduction speech signal and a noise category; A noise intensity obtaining module, configured to obtain a noise intensity estimation value according to the first noise reduction speech signal and the noisy speech signal; The noise reduction speech acquisition module is configured to perform signal mixing on the first noise reduction speech signal and the noisy speech signal according to the noise intensity estimation value and the noise category to obtain a second noise reduction speech signal.

Images