Speech noise reduction method, device, equipment and storage medium

Abstract

The present application discloses a method for speech denoising, comprising: obtaining multiple channels of noisy speech; synchronously collecting the multiple channels of noisy speech; extracting feature data of the multiple channels of noisy speech; calling a network model to process the feature data of the noisy speech to obtain a masking value of the speech, and using the masking value of the speech to calculate the cross-power spectral density of a steering vector and noise; using the cross-power spectral density of the steering vector and the noise to process the noisy speech to obtain the denoised speech, so as to improve the effect of denoising the noisy speech.

Description

Technical Field

[0001] The present application relates to the technical field of speech noise reduction methods, and more specifically, to a speech noise reduction method, device, equipment and storage medium. Background Art

[0002] Currently, speech noise reduction technology is a technology that extracts target speech from noisy speech.

[0003] MVDR is one of the most widely used beamforming technologies. Its principle is to try to minimize the power of the beamforming signal while maintaining unit gain in the desired direction. For the dual-channel noise reduction algorithm, the cross-power spectral density of the steering vector and the noise is calculated through the received signals of the dual microphones, and the cross-power spectral density of the steering vector and the noise is processed according to the formula to calculate the MVDR coefficient (which can be understood as a weight coefficient or beam coefficient), and then the MVDR coefficient is applied to the received signals of the dual microphones to obtain the output after beamforming.

[0004] Since the beamforming method is obtained by linearly weighting the received signals of the dual microphones after estimating the weight coefficient, it is equivalent to a linear speech noise reduction technology. This makes the beamforming operation basically distortion-free for the target speech, and can achieve better noise reduction effects when the signal-to-noise ratio is relatively high.

[0005] However, when the signal-to-noise ratio of the dual microphones' received signals is low, the cross-power spectral density of the steering vector and the noise is inaccurately calculated, making it difficult to accurately calculate the MVDR coefficient, which weakens the noise suppression capability. In addition, beamforming adjusts the MVDR coefficient so that the beam can suppress noise in different directions from the target speech (which can be understood as the standard speech in the microphone's received signal, i.e., speech that is not noise), but it cannot suppress noise in the same direction as the target speech. Summary of the invention

[0006] In view of the above problems, the present application provides a method, apparatus, device and storage medium for speech noise reduction to improve the effect of noise reduction processing on noisy speech. The specific scheme is as follows:

[0007] In a first aspect of the present application, a method for speech noise reduction is provided, comprising:

[0008] Acquire multiple channels of noisy speech; synchronously acquire multiple channels of noisy speech;

[0009] Extracting feature data of multiple noisy speech;

[0010] Call the network model to process the feature data of the noisy speech and obtain the masking value of the speech;

[0011] The cross power spectrum density between the steering vector and the noise is calculated using the masking value of the speech;

[0012] The cross power spectral density of the steering vector and the noise is used to process the noisy speech and obtain the denoised speech.

[0013] Optionally, extracting feature data of multiple channels of noisy speech includes:

[0014] Extract each noisy speech to obtain the speech features of the noisy speech;

[0015] Calculate the difference characteristics of the inter-channel signals of multi-channel noisy speech.

[0016] Optionally, extracting each noisy speech to obtain speech features of the noisy speech includes:

[0017] Perform short-time Fourier transform on each noisy speech to obtain the speech features of the noisy speech.

[0018] Optionally, the difference characteristics of the inter-channel signals of the multi-channel noisy speech include: level difference, time difference, phase difference and mutual correlation coefficient.

[0019] Optionally, using the cross power spectral density of the steering vector and the noise to process the noisy speech to obtain the noise-reduced speech includes:

[0020] Process the cross power spectral density of the steering vector and the noise to obtain the beam coefficient;

[0021] The beam coefficients are used to perform beamforming on the noisy speech to obtain a beam output signal, which is used as the denoised speech.

[0022] Optionally, beamforming is performed on multiple noisy speech paths using beam coefficients to obtain beam output signals, including:

[0023] The beam coefficients are used to perform beamforming on a noisy speech channel to obtain a beam output signal.

[0024] Optionally, it also includes:

[0025] Performing short-time Fourier transform on the beam output signal to obtain characteristic data of the beam output signal;

[0026] Calling the network model to process the characteristic data of the beam output signal to obtain the masking value of the beam output signal;

[0027] The mask value of the beam output signal is multiplied by the beam output signal to obtain an updated beam output signal.

[0028] In a second aspect of the present application, a device for speech noise reduction is provided, comprising:

[0029] An acquisition unit is used to acquire multiple channels of noisy speech; multiple channels of noisy speech are acquired synchronously;

[0030] An extraction unit, used for extracting feature data of multiple channels of noisy speech;

[0031] A calling unit, used for calling a network model to process feature data of noisy speech to obtain a masking value of the speech;

[0032] A calculation unit, used to calculate the cross power spectrum density of the steering vector and the noise by using the masking value of the speech;

[0033] The processing unit is used to process the noisy speech by using the cross power spectrum density of the steering vector and the noise to obtain the noise-reduced speech.

[0034] A third aspect of the present application provides a speech noise reduction device, including: a memory and a processor;

[0035] Memory, used to store programs;

[0036] The processor is used to execute the program to implement the various steps of the above method for speech noise reduction.

[0037] In a fourth aspect of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned method for speech noise reduction is implemented.

[0038] By means of the above technical scheme, in the method for speech noise reduction provided by the present application, a network model is used to estimate the characteristic data of noisy speech to obtain the masking value of the speech, which can ensure that the cross-power spectral density of the steering vector and the noise calculated by the masking value of the speech is more accurate, so as to more accurately realize the beamforming of multi-path noisy speech and obtain better noise reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0040] Figure 1 A schematic diagram of a network model training process provided in an embodiment of the present application;

[0041] Figure 2 A flow chart of a method for speech noise reduction provided in an embodiment of the present application;

[0042] Figure 3 A spectrogram of clean speech provided in an embodiment of the present application;

[0043] Figure 4 A spectrogram of noisy speech provided in an embodiment of the present application;

[0044] Figure 5 A spectrogram of the audio output by the MVDR beamforming algorithm provided in an embodiment of the present application;

[0045] Figure 6 A spectrogram of the audio output by the FSMN model provided in the embodiment of the present application;

[0046] Figure 7 A spectrogram of the beam output audio after the FSMN network calculates the MVDR coefficients provided in the embodiment of the present application;

[0047] Figure 8 A spectrogram of the output audio of the FSMN network provided in the embodiment of the present application that calculates the MVDR coefficients for beamforming and post-filtering;

[0048] Fig. 9 A schematic diagram of the structure of the speech noise reduction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] Speech noise reduction technology is a technology that extracts target speech from noisy speech. Speech noise reduction technology is divided into single-channel algorithms and multi-channel algorithms according to whether the number of channels used by the microphone to receive the signal is 1. Among them, although the single-channel noise reduction algorithm has the characteristics of low cost and low structural design requirements, it generally only performs well under stable noise and has certain limitations on usage scenarios. The speech collected by an array composed of multiple microphones can contain richer spatial features and can provide additional information for speech noise reduction.

[0051] The existing dual-channel speech noise reduction can be roughly divided into linear methods based on beamforming and nonlinear methods based on deep learning. Among them, the minimum variance distortionless response (MVDR) algorithm is an adaptive beamforming algorithm based on the maximum signal-to-noise ratio (SINR) criterion. It can adaptively minimize the array output power while maximizing the signal-to-noise ratio in the desired direction. It is the most commonly used beamforming technology.

[0052] MVDR is one of the most widely used beamforming technologies. Its principle is to try to minimize the power of the beamforming signal while maintaining unit gain in the desired direction. For the dual-channel noise reduction algorithm, the cross-power spectral density of the steering vector and the noise is calculated through the received signals of the dual microphones, and the cross-power spectral density of the steering vector and the noise is processed according to the formula to calculate the MVDR coefficient (also called weight coefficient or beam coefficient), and then the MVDR coefficient is applied to the received signals of the dual microphones to obtain the output after beamforming.

[0053] Since the beamforming method is obtained by linearly weighting the received signals of the dual microphones after estimating the weight coefficient, it is equivalent to a linear speech noise reduction technology. This makes the beamforming operation basically distortion-free for the target speech, and can achieve better noise reduction effects when the signal-to-noise ratio is relatively high.

[0054] However, when the signal-to-noise ratio of the dual microphones' received signals is low, the cross-power spectral density of the steering vector and the noise is inaccurately calculated, making it difficult to accurately calculate the MVDR coefficient, which weakens the noise suppression capability. In addition, beamforming adjusts the MVDR coefficient so that the beam can suppress noise in different directions from the target speech (which can be understood as the standard speech in the microphone's received signal, i.e., speech that is not noise), but it cannot suppress noise in the same direction as the target speech.

[0055] Another type of dual-channel noise reduction method based on deep learning has developed rapidly in recent years. The powerful nonlinear representation ability of deep neural networks enables the model to deeply abstract features. Similar to other speech applications using deep learning, the main influencing factors include acoustic features, training objectives, and network structure. Unlike single-channel deep learning noise reduction, the extracted features need to include the speech features of each channel and the difference features of the signals between channels. The speech features of each channel can be obtained by calculating the amplitude spectrum of each channel signal after short-time Fourier transform (STFT); the difference features of the signals between channels can be obtained by calculating the level difference, time difference, phase difference, and mutual correlation coefficient. The difference features of the signals between channels can well represent the spatial orientation information of the target speech and the microphone, so as to further combine the time-frequency domain features of each channel to reduce the noise of the speech.

[0056] In addition, the output signal of the network model is generally to calculate the masking values ​​such as IBM (Ideal binary mask), IRM (Ideal ratio mask), SMM (spectral magnitude mask) on each time-frequency unit, and can also use cIRM (complex ideal ratio mask), PSM (Phase Sensitive Mask) and other masking values. Multiplying the masking value with the power spectrum of the noisy speech can obtain the power spectrum of the clean speech, thereby restoring the denoised speech.

[0057] Different network structures may also have different degrees of impact on the noise reduction performance. Currently, commonly used network structures include convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), etc.

[0058] The dual-channel noise reduction method based on deep learning also has certain shortcomings. Taking the deep learning method based on time-frequency masking as an example, the network model estimates the masking value of each time-frequency unit to determine the probability of the speech signal appearing at the current time-frequency point, and multiplies the masking value with the noisy speech power spectrum to obtain the signal power spectrum after noise reduction. Since the time-frequency unit will be disturbed by noise, the masking value calculated by the network model is not accurate, resulting in a certain degree of distortion or even sound loss in the signal after noise reduction, which will greatly reduce the speech intelligibility and speech quality. In one example, the frequency of the noise and the target speech is similar. When the network model processes the input data, the noise will interfere with the processing process of the network model.

[0059] In order to solve the above problems, an embodiment of the present application proposes a method for speech noise reduction, which can be applied to a terminal with data processing capabilities, and the terminal can be a mobile phone, a computer, a server, a cloud, etc.

[0060] The method for speech noise reduction provided in this embodiment can be understood as a dual-microphone speech noise reduction method that combines beamforming and deep learning. The network model is used to estimate the steering vector of the received signal of the dual microphones and the cross-power spectral density of the noise to more accurately calculate the weight coefficients for beamforming. The beam coefficients are then used to beamform the received signal of the dual microphone signal to obtain a distortion-free response in the desired direction and suppress interference and noise in other directions to obtain an output beamformed signal. Finally, the signal-to-noise ratio of the current received signal can be determined, and the post-filtering of the neural network model can be selectively performed to further suppress noise.

[0061] Combining the above two methods can improve the situation where the beam coefficient estimation is inaccurate and the noise suppression is insufficient when the signal-to-noise ratio is low; it can also improve the distortion of the speech caused by directly using the neural network model when the signal-to-noise ratio is high, thereby achieving better noise reduction effect.

[0062] First of all, it should be noted that before executing the method for speech noise reduction provided in the embodiment of the present application, it is necessary to first build a network model based on deep learning.

[0063] In this embodiment, the network model uses a feedforward sequential memory neural network (FSMN). The FSMN network improves the traditional DNN network by adding a memory module to the hidden layer to store node information at past moments.

[0064] The network model includes: input layer, hidden layer and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0065] It can be understood that the hidden layer includes: 3 convolutional layers connected in series, the FSMN layer with a memory module (also called a convolutional layer with a memory module), and a batch normalization layer (Batch Normalization, BN). When the FSMN layer with a memory module runs, it needs to use the data of the memory module.

[0066] In the network model, the output data of the three convolutional layers connected in series are provided to the FSMN layer with a memory module, and the output data of the FSMN layer with a memory module is provided to the batch normalization layer. The output data of the batch normalization layer is output through the output layer. The output data of the batch normalization layer can be mask values ​​such as IBM (Ideal binary mask), IRM (Ideal Ratio Mask), SMM (Spectral magnitude mask), or cIRM (Complex Ideal Ratio Mask), PSM (Phase Sensitive Mask), etc. The following is an example of output data cIRM.

[0067] The process of training the network model, see Figure 1 , including the steps of:

[0068] S101. Construct training data.

[0069] A large amount of clean speech and noise is collected, where clean speech can be understood as speech that is not pure and usually includes some noise. The collected noise is used to synthesize training data, which may include scattered noise and point source noise.

[0070] Clean speech is processed by the mirror source method to simulate room impulse responses of different scenarios. It can be understood that: for the clean speech of one channel (which can also be understood as a clean signal), the clean speech is processed by the mirror source method to obtain the room impulse response of at least two channels of speech. In this embodiment, the room impulse response of two channels of speech is obtained as an example for explanation.

[0071] Voice Activity Detection (VAD) processes the clean speech to obtain the VAD-processed clean speech s(n).

[0072] The clean speech s(n) after VAD processing is convolved with the room impulse response h(n) of each speech channel to obtain a dual-channel speech signal.

[0073] The dual-channel speech signals are respectively superimposed with noise noise(n) to obtain dual-channel noisy speech training data y(n)=s(n)*h(n)+noise(n). The dual-channel speech signals and noise can be mixed according to a certain signal-to-noise ratio, and the signal-to-noise ratio used when the speech signal and noise of each channel of the dual-channel speech signal are mixed needs to be the same.

[0074] It should also be noted that each training data is also configured with a marking result, and the marking result includes: an ideal cIRM value of the training result.

[0075] S102: Extract feature data of training data.

[0076] Perform short-time Fourier transform on the noisy speech training data of each channel. The real and imaginary parts of the short-time Fourier transform data of the dual-channel speech are used as the speech features of each channel.

[0077] The phase difference (IPD) of the second channel relative to the first channel is calculated as the difference feature of the inter-channel signal, and the difference feature of the inter-channel signal is used to characterize the spatial information.

[0078] The speech features of each channel and the difference features of the signals between channels are used as input data of the network model.

[0079] S103: Obtain the original model of the network model.

[0080] The original model of the network model is as shown in the above content, and a feedforward sequence memory neural network can be used. In addition, the structure of the network model can also be as shown in the above content, which will not be described here.

[0081] S104: Use the original model of the network model to process the feature data of the training data to obtain a detection result cIRM.

[0082] S105, using the weighted amplitude phase loss function to calculate the loss value of the detection result cIRM and the labeling result of each training data, to obtain the loss value of the network model.

[0083] In the training of the network model, the cIRM of the speech is calculated so that the network model can use the amplitude and phase information of the speech at the same time. When the loss function adopts the MSE (mean-square error) loss function, it is equivalent to calculating the Euclidean distance between the estimated speech and the ideal speech in the complex plane composed of phase and amplitude, so that the network model gives the same weight to phase and amplitude.

[0084] However, most of the structure of noisy speech comes from amplitude information, so the accuracy of amplitude estimation is more important in the noise reduction process. Therefore, a weighted amplitude-phase loss function is used in this embodiment, as shown in Formula 1 below.

[0085]

[0086] Among them, M(t, f) and represent the ideal cIRM value and the estimated cIRM value at time frame t and frequency point f, respectively, θ M and is the corresponding phase, and N represents the number of inputs involved in the loss function calculation. The first part of the formula is the traditional MSE loss function, and the second part is the weighted phase estimation loss function term. When the phase difference between the estimated speech and the ideal speech is close to 0, the loss function decreases, and conversely, when the phase difference is close to -π or π, the loss function increases.

[0087] In order to prevent the phase estimation weight from affecting the accuracy of the network's estimated amplitude spectrum, the phase estimation term is weighted using a weight factor α, for example, α = 0.1. In addition, the phase difference is multiplied by the amplitude information of cIRM, so that the influence of the phase is increased when the speech amplitude is large.

[0088] Of course, other loss functions can also be used.

[0089] S106: Determine whether the loss value of the network model meets the convergence condition of the network model.

[0090] In some embodiments, the network model convergence condition may be that the loss value of the model is less than or equal to a preset loss threshold. In other words, the loss value of the model may be compared with the loss threshold. If the loss value of the model is greater than the loss threshold, it may be determined that the loss value of the model does not meet the model convergence condition. Conversely, if the loss value of the model is less than or equal to the loss threshold, it may be determined that the loss value of the model meets the model convergence condition.

[0091] If the loss value of the network model meets the convergence condition of the model, it means that the network model training is completed. The trained model can be used in the speech denoising method proposed in the following embodiment to process the noisy speech input to the network model to obtain cIRM.

[0092] If the loss value of the network model does not meet the convergence condition of the network model, step S107 is executed to calculate the parameter update value of the network model according to the loss value of the network model, and the network model is updated with the parameter update value of the network model. The updated network model is used to return to step S104. The updated network model is used to continue processing the feature data of the training data, obtain the detection result, and continue to execute the subsequent process until the loss value of the network model meets the convergence condition of the network model.

[0093] Based on the above-mentioned trained network model, this embodiment provides a method for speech noise reduction, see Figure 2 , including the steps of:

[0094] S201: Acquire reception signals of dual microphones.

[0095] Among them, the received signals of the dual microphones can be understood as two-channel noisy speech, that is, two-way noisy speech signals. Of course, the dual microphones need to run at the same time to synchronously obtain the noisy speech of the two channels.

[0096] S202: Extract feature data of the signals received by the dual microphones.

[0097] The feature data includes the speech features of the noisy speech of each channel and the difference features of the signals between channels. The speech features of the noisy speech of each channel can be obtained by calculating the short-time Fourier transform (STFT) of the noisy speech of each channel. The difference features of the signals between channels can be calculated by level difference, time difference, phase difference and mutual correlation coefficient.

[0098] S203: Call the network model to process the feature data of the received signals of the dual microphones to obtain the masking value of the speech.

[0099] S204: Calculate the cross-power spectral density of the steering vector and the noise using the masking value of the speech.

[0100] After getting the masking value cIRM of the speech x (t, f), the noise masking value cIRMn(t, f) can be calculated by the following formula 2.

[0101] Formula 2

[0102] cIRMn(t,f)=1-cIRM x (t, f)

[0103] According to the following formula 3, the cross power spectrum density Φ of speech is obtained: xx (f) and the cross power spectral density of the noise Φ nn (f). Where Y(t, f) represents the matrix composed of the short-time Fourier transform of the dual-channel received signal, Y H (t, f) represents the conjugate transpose of the matrix composed of the short-time Fourier transform of the dual-channel received signals, and T refers to the total number of frames of the noisy speech after the short-time Fourier transform.

[0104] Formula 3

[0105]

[0106]

[0107] The steering vector d(f) can be obtained from the cross power spectrum density Φ of the speech xx (f) is calculated as shown in Formula 4 below.

[0108] Formula 4

[0109] d(f)=P{Φ xx (f)}

[0110] S205: Process the cross-power spectral density of the steering vector and the noise to obtain a beam coefficient.

[0111] When the MVDR beamforming method is used, the beam coefficient can be obtained by the cross power spectrum density Φ of the steering vector d(f) and the noise. nn (f) is calculated as shown in the following formula 5. Where, d H (f) is the conjugate transpose of the steering vector.

[0112]

[0113] S206 : Use the beam coefficients to perform beamforming on the received signals of the dual microphones to obtain beam output signals.

[0114] The beam output signal is obtained by multiplying the beam coefficient with the amplitude spectrum of the received signal of one microphone, as shown in the following formula 6. Of course, one of the received signals of the two microphones can be randomly used. The beam output signal can be understood as the speech after noise reduction.

[0115]

[0116] It should also be noted that if post-filtering of the beam output signal is required, a trained neural network can be used for further noise reduction.

[0117] In a possible embodiment, the signal-to-noise ratio of the beam output signal is calculated, and it is determined whether the signal-to-noise ratio of the beam output signal is less than a threshold. In an example, the threshold may be 0, -10, etc.

[0118] If the signal-to-noise ratio of the beam output signal is determined to be less than the threshold, a short-time Fourier transform (STFT) is performed on the beam output signal to obtain characteristic data of the beam output signal, and the network model is called to process the characteristic data of the beam output signal to obtain a mask value of the beam output signal. The mask value of the beam output signal is then multiplied by the beam output signal to obtain an updated beam output signal. If the signal-to-noise ratio of the beam output signal is determined to be not less than the threshold, no processing is performed.

[0119] In this embodiment, the judgment is made by calculating the signal-to-noise ratio of the beam output signal. If the signal-to-noise ratio is very low, the beam output signal can be filtered again through the neural network to further suppress the noise that the beam cannot eliminate, such as non-stationary noise and noise in the same direction as the target, so as to improve the noise reduction capability in a low signal-to-noise ratio environment.

[0120] It should also be noted that if the network model requires two input signals, the feature data of the beam output signal and one noisy speech can be extracted separately and used as the input signal of the network model for noise reduction processing.

[0121] In order to verify the speech noise reduction performance of the speech noise reduction method provided in the embodiment of the present application, and its advantages over the beamforming method and the deep learning method alone, 100 hours of simulated dual-channel noisy speech was used for FSMN model training. The input features were the amplitude spectra and phase differences of the two microphone signals. The training target was the cIRM value of the microphone signal of the first channel. The spectrogram of the clean speech selected during the test is as follows: Figure 3 As shown, the spectrogram of the noisy speech received by the first channel microphone is as follows Figure 4 As shown in the figure, the spectrogram of the noisy speech shows that the signal-to-noise ratio of the selected test audio is relatively low, and in addition to the background noise, it also contains some speech interference components.

[0122] The results of MVDR beamforming on the two input signals of the test speech are as follows: Figure 5By comparing the spectrogram of noisy speech, it can be found that the mid- and high-frequency background noise is suppressed to a certain extent after MVDR, but due to the low signal-to-noise ratio of the test audio, the estimated coefficient of beamforming is not accurate, so a good noise reduction effect cannot be achieved.

[0123] The result of directly using FSMN network noise reduction on the test speech is as follows Figure 6 As shown in the spectrogram, it can be seen that the network suppresses noise very well, but some speech structures are also eliminated, resulting in distortion, which greatly reduces the speech intelligibility and speech quality.

[0124] The method provided in this embodiment is used for the test speech. That is, the FSMN network is first used to estimate the cIRM values ​​of the speech and noise, and then the cross power spectrum density and steering vector of the noise are calculated to calculate the MVDR beam coefficient. The final beam output signal is as follows: Figure 7 As shown. Figure 4 Compared with directly using MVDR, the effect of calculating the beam coefficient after estimating the speech and noise related parameters through deep learning is much better. The structure of the speech part is clearer, and the interfering speech and background noise are suppressed to a certain extent, and the overall speech quality is better.

[0125] In addition, since the signal-to-noise ratio after beam output is still low, the same FSMN network is used for post-filtering, and the results are as follows: Figure 8 As shown, Figure 8 The results compared to Figure 6 The speech structure is well preserved, compared to Figure 7 It suppresses more noise and interference, and the overall noise reduction effect is relatively the best.

[0126] The perceptual evaluation of speech quality (PESQ) values ​​of the output speech obtained by the aforementioned methods are calculated, as shown in Table 1.

[0127] Noise Reduction Algorithm noisy MVDR FSMN FSMN-MVDR FSMN-MVDR-FSMN PESQ value 1.226 1.232 1.137 1.424 2.291

[0128] It can be seen from Table 1 that the results are basically consistent with the conclusions of the spectrogram. The dual-microphone speech noise reduction method based on the combination of beamforming and deep learning provided in this embodiment can well improve the speech quality of noisy speech, and is greatly improved compared to the noise reduction algorithm using beamforming or deep learning alone. In addition, the use of deep learning post-filtering under low signal-to-noise ratio conditions can further improve the noise reduction performance, greatly improving the speech quality.

[0129] Another embodiment of the present application provides a device for speech noise reduction, including:

[0130] An acquisition unit is used to acquire multiple channels of noisy speech; the multiple channels of noisy speech are acquired synchronously;

[0131] An extraction unit, used for extracting feature data of the multiple channels of noisy speech;

[0132] A calling unit, used for calling a network model to process the feature data of the noisy speech to obtain a masking value of the speech;

[0133] A calculation unit, configured to calculate a cross-power spectral density of a steering vector and noise using a masking value of the speech;

[0134] The processing unit is used to process the noisy speech by using the cross power spectrum density of the steering vector and the noise to obtain the denoised speech.

[0135] In the speech noise reduction device provided in this embodiment, the specific implementation process of each unit may be the same as the content of the aforementioned method embodiment, and will not be repeated here.

[0136] The speech noise reduction device provided in the embodiment of the present application can be applied to speech noise reduction equipment, such as terminals: mobile phones, computers, etc. Optionally, Figure 6 The hardware structure diagram of the speech noise reduction device is shown in FIG. Figure 6 , the hardware structure of the speech noise reduction device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0137] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0138] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0139] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0140] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0141] Acquire multiple channels of noisy speech; synchronously acquire multiple channels of noisy speech;

[0142] Extracting feature data of multiple noisy speech;

[0143] Call the network model to process the feature data of the noisy speech and obtain the masking value of the speech;

[0144] The cross power spectrum density between the steering vector and the noise is calculated using the masking value of the speech;

[0145] The cross power spectral density of the steering vector and the noise is used to process the noisy speech and obtain the denoised speech.

[0146] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0147] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0148] Acquire multiple channels of noisy speech; synchronously acquire multiple channels of noisy speech;

[0149] Extracting feature data of multiple noisy speech;

[0150] Call the network model to process the feature data of the noisy speech and obtain the masking value of the speech;

[0151] The cross power spectrum density between the steering vector and the noise is calculated using the masking value of the speech;

[0152] The cross power spectral density of the steering vector and the noise is used to process the noisy speech and obtain the denoised speech.

[0153] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0154] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0155] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0156] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for speech noise reduction, characterized in that: include: Acquire two channels of noisy speech; the two channels of noisy speech are acquired synchronously; Extracting feature data of the two-channel noisy speech; Calling a network model to process the feature data of the noisy speech to obtain a masking value of the speech; Using the masking value of the speech, a cross-power spectral density of the steering vector and the noise is calculated; Processing the cross power spectral density of the steering vector and the noise to obtain a beam coefficient; Performing beamforming on the noisy speech using the beam coefficient to obtain a beam output signal; Calculating the signal-to-noise ratio of the beam output signal, and determining whether the signal-to-noise ratio of the beam output signal is less than a threshold value; if it is determined that the signal-to-noise ratio of the beam output signal is less than the threshold value, performing a short-time Fourier transform on the beam output signal to obtain feature data of the beam output signal, and extracting feature data of one noisy speech to obtain feature data of one noisy speech; calling the network model to process the feature data of the beam output signal and the feature data of the one noisy speech to obtain a masking value of the beam output signal; and multiplying the masking value of the beam output signal by the beam output signal to obtain an updated beam output signal; Wherein, the network model adopts a feedforward sequence memory neural network; The network model is trained in the following manner: constructing training data, extracting feature data of the training data, obtaining the original model of the network model, using the original model of the network model to process the feature data of the training data, and obtaining a detection result; using a weighted amplitude-phase loss function to calculate the loss of the detection result and the labeling result of each training data to obtain a loss value of the network model; judging whether the loss value of the network model meets the convergence condition of the network model. If the loss value of the network model meets the convergence condition, it indicates that the training of the network model is completed.

2. The method for speech noise reduction according to claim 1, characterized in that: The step of extracting feature data of the two channels of noisy speech includes: Extract each noisy speech to obtain the speech features of the noisy speech; Calculate the difference characteristics of the inter-channel signals of two noisy speech channels.

3. The method for speech noise reduction according to claim 2, characterized in that: The step of extracting each noisy speech to obtain speech features of the noisy speech includes: Performing short-time Fourier transform on each noisy speech channel to obtain speech features of the noisy speech.

4. The method for speech noise reduction according to claim 2, characterized in that: The difference characteristics of the inter-channel signals of the two-channel noisy speech include: level difference, time difference, phase difference and mutual correlation coefficient.

5. The method for speech noise reduction according to claim 1, characterized in that: The using the beam coefficient to perform beamforming on the noisy speech to obtain a beam output signal includes: The beam coefficient is used to perform beamforming on one channel of the noisy speech to obtain the beam output signal.

6. A speech noise reduction device, characterized in that: include: An acquisition unit, used for acquiring two-channel noisy speech; the two-channel noisy speech is acquired synchronously; An extraction unit, used to extract feature data of the two channels of noisy speech; A calling unit, used for calling a network model to process the feature data of the noisy speech to obtain a masking value of the speech; wherein the network model adopts a feedforward sequence memory neural network; the network model is trained in the following manner: constructing training data, extracting feature data of the training data, obtaining an original model of the network model, and using the original model of the network model to process the feature data of the training data to obtain a detection result; using a weighted amplitude phase loss function to calculate the loss of the detection result and the labeling result of each training data to obtain a loss value of the network model; judging whether the loss value of the network model meets the convergence condition of the network model, and if the loss value of the network model meets the convergence condition, it means that the training of the network model is completed; A calculation unit, configured to calculate a cross-power spectral density of a steering vector and noise using a masking value of the speech; a processing unit, configured to process a cross power spectrum density of the steering vector and the noise to obtain a beam coefficient; Performing beamforming on the noisy speech using the beam coefficient to obtain a beam output signal; Calculate the signal-to-noise ratio of the beam output signal, and determine whether the signal-to-noise ratio of the beam output signal is less than a threshold value; if it is determined that the signal-to-noise ratio of the beam output signal is less than the threshold value, perform short-time Fourier transform on the beam output signal to obtain feature data of the beam output signal, and extract feature data of one noisy speech to obtain feature data of one noisy speech; call the network model to process the feature data of the beam output signal and the feature data of the one noisy speech to obtain a masking value of the beam output signal; multiply the masking value of the beam output signal by the beam output signal to obtain an updated beam output signal.

7. A speech noise reduction device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the speech noise reduction method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech noise reduction method according to any one of claims 1 to 5 is implemented.

Images