Noise reduction method and device for non-negative matrix factorization and deep learning, medium and equipment
By performing non-negative matrix decomposition on the subband and using deep learning to correct the noise reduction gain, combining the advantages of non-negative matrix decomposition and deep learning, the problems of insufficient speech noise reduction performance and difficulty in deploying deep learning models in the prior art are solved. , achieving efficient voice noise reduction effect.
Patent Information
- Application Number
- CN202411538440.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-07
AI Technical Summary
The prior art has a fair performance in speech noise reduction at lower signal-to-noise ratios, and deep learning models are difficult to deploy in embedded systems, especially in Bluetooth-related devices.
Combining non-negative matrix decomposition and deep learning, voice noise reduction is achieved by performing non-negative matrix decomposition on the subband, and correcting the noise reduction gain based on deep learning.
It realizes efficient speech noise reduction at a lower signal-to-noise ratio to ensure sound quality, while avoiding the shortcomings of non-negative matrix decomposition and deep learning models, and reducing the amount of computing and storage.
Smart Images
Figure CN119400198A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of Bluetooth audio technology, and in particular, relates to a speech noise reduction method, device, storage medium and equipment combining non-negative matrix decomposition and deep learning. Background Art
[0002] In many Bluetooth applications, such as Bluetooth calls, Bluetooth microphones, and recordings, noise reduction is required.
[0003] Nonnegative Matrix Factorization (NMF) makes all matrix components non-negative and achieves nonlinear dimensionality reduction. NMF has gradually become one of the most popular multidimensional data processing tools in research fields such as signal processing, biomedical engineering, pattern recognition, computer vision and image engineering.
[0004] Non-negative matrix factorization has certain applications in speech noise reduction, but its performance is average at low signal-to-noise ratios and can cause distortion.
[0005] Deep learning has received widespread attention in speech noise reduction. Currently, many mature algorithms have been successfully deployed on servers and PCs with good results. However, due to the complexity of the network model and the large memory space occupied by the weights, it is difficult to deploy in embedded systems. It usually requires the assistance of NPU. Many Bluetooth-related devices are based on embedded systems, and few of them are equipped with NPU. Summary of the invention
[0006] In response to the above-mentioned technical problems existing in the prior art, the present application provides a speech denoising method, apparatus, storage medium and device combining non-negative matrix decomposition and deep learning. Based on the MDCT spectral coefficients, the noise reduction gain is generated by performing non-negative matrix decomposition on the sub-band, and then the noise reduction gain is corrected based on deep learning, thereby achieving speech denoising. It can not only ensure the sound quality, but also reduce the size of the matrix, avoiding the shortcomings of non-negative matrix decomposition and avoiding the use of overly complex network models for deep learning.
[0007] In order to achieve the above-mentioned purpose, the first technical solution adopted in the present application is: to provide a speech denoising method combining non-negative matrix decomposition and deep learning, including: inputting monophonic noisy speech PCM data and performing discrete cosine transform to obtain MDCT spectrum coefficients; calculating subband energy according to MDCT spectrum coefficients and constructing a subband energy observation matrix; performing non-negative matrix decomposition on the subband energy observation matrix and calculating speech subband energy values and noise subband energy values; calculating the first subband denoising gain according to the speech subband energy values and noise subband energy values; inputting the first subband denoising gain into a pre-trained deep neural network, outputting the second subband denoising gain, and performing interpolation on the second subband denoising gain to obtain all frequency bin gains; obtaining denoising spectrum coefficients according to the frequency bin gains and MDCT spectrum coefficients; and performing inverse discrete cosine transform and overlap-addition on the denoising spectrum coefficients to output denoised speech PCM.
[0008] The second technical solution adopted in the present application is: to provide a speech denoising device combining non-negative matrix decomposition and deep learning, including: a module for inputting monophonic noisy speech PCM data and performing discrete cosine transform to obtain MDCT spectrum coefficients; a module for calculating subband energy according to MDCT spectrum coefficients and constructing a subband energy observation matrix; a module for performing non-negative matrix decomposition on the subband energy observation matrix and calculating speech subband energy values and noise subband energy values; a module for calculating the first subband denoising gain according to the speech subband energy values and the noise subband energy values; a module for inputting the first subband denoising gain into a pre-trained deep neural network, outputting the second subband denoising gain, and performing interpolation on the second subband denoising gain to obtain all frequency bin gains; a module for obtaining denoising spectrum coefficients according to frequency bin gains and MDCT spectrum coefficients; a module for performing inverse discrete cosine transform and overlap-addition on the denoising spectrum coefficients to output denoised speech PCM.
[0009] The third technical solution adopted in the present application is: providing a computer-readable storage medium storing computer instructions, wherein the computer instructions are operated to execute the speech denoising method combining non-negative matrix decomposition and deep learning in solution one.
[0010] The fourth technical solution adopted in the present application is: to provide a computer device, which includes a processor and a memory, the memory stores computer instructions, wherein the processor operates the computer instructions to execute the speech denoising method combining non-negative matrix decomposition and deep learning in solution one.
[0011] The beneficial effects that can be achieved by the technical solution of the present application are: the technical solution of the present application can be applied to both classic Bluetooth (BR, EDR) and low-power Bluetooth (BLE), based on MDCT spectrum coefficients, by performing non-negative matrix decomposition on sub-bands to generate noise reduction gain, and then correcting the noise reduction gain based on deep learning, thereby achieving speech noise reduction, ensuring sound quality, avoiding the shortcomings of non-negative matrix decomposition, and avoiding the use of overly complex network models for deep learning. Based on the processing of discrete cosine transform, no phase spectrum is required during inverse transformation. Compared with the traditional Fourier transform and inverse transform noise reduction that requires phase spectrum, it avoids the damage to the reconstructed speech due to the inability to obtain the phase spectrum of pure speech and only using the phase spectrum of noisy speech; performing non-negative matrix decomposition on sub-bands, compared with the traditional amplitude spectrum, the size of the matrix is greatly reduced, and the amount of calculation and storage are correspondingly reduced; the deep learning model is mainly used to correct the noise reduction gain of non-negative matrix decomposition. Compared with the traditional end-to-end deep learning noise reduction model based on amplitude spectrum, the network structure is simpler, and the amount of calculation and storage are correspondingly reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 It is a flowchart of a specific implementation method of the speech denoising method combining non-negative matrix decomposition and deep learning in the present application;
[0013] Figure 2 is a schematic diagram of a specific example of the training process of the pre-trained deep neural network in this application;
[0014] Figure 3 It is a schematic diagram of a specific example of speech denoising combining non-negative matrix decomposition and deep learning in the present application;
[0015] Figure 4 It is a schematic diagram of a specific implementation of the speech noise reduction device of the present application that combines non-negative matrix decomposition and deep learning. DETAILED DESCRIPTION
[0016] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present application can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present application.
[0017] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0018] The specific implementation of the present application is described in detail below using the 48 kHz sampling rate as an example. The principles of other sampling rates are similar to those of the 48 kHz sampling rate and will not be repeated here.
[0019] Figure 1 It is a flowchart of a specific implementation method of the speech denoising method combining non-negative matrix decomposition and deep learning in the present application.
[0020] exist Figure 1 In a specific implementation shown, the speech denoising method combining non-negative matrix decomposition and deep learning of the present application includes process S101, inputting monophonic noisy speech PCM data and performing discrete cosine transform to obtain MDCT spectrum coefficients.
[0021] In a specific embodiment of the present application, monophonic noisy speech PCM data is input and discrete cosine transform is performed to obtain MDCT spectrum coefficients, including: framing the monophonic noisy speech PCM data with a fixed frame length and windowing the obtained audio frames; and discrete cosine transform is performed on the windowed audio frames to obtain MDCT spectrum coefficients.
[0022] Specifically, first, input the mono-channel noisy speech PCM data, then perform frame division and windowing on the mono-channel noisy speech PCM data, and then perform discrete cosine transform on the windowed audio frame to obtain the MDCT spectrum coefficient. Among them, the frame division is to divide the input PCM sampling data into fixed frame lengths, and process one frame at a time. Taking the 48kHz sampling rate as an example, the typical frame length can be 10ms, corresponding to 480 sampling points; the execution process of windowing and MDCT spectrum coefficients is as follows:
[0023] t(n)=x(ZN F +n), for n=0...2·N F -1-Z
[0024] t(2N F-Z+n)=0,for n=0...Z-1
[0025]
[0026] Where x(n) is the input audio PCM signal, w A (n) is the analysis window, Z is the number of leading 0s in the analysis window, and N F is the size of the data frame processed each time, and X(k) is the MDCT spectrum coefficient.
[0027] exist Figure 1 In a specific implementation shown, the speech denoising method combining non-negative matrix decomposition and deep learning of the present application includes process S102, calculating subband energy according to MDCT spectral coefficients and constructing a subband energy observation matrix.
[0028] In a specific embodiment of the present application, subband energy is calculated based on MDCT spectrum coefficients, including: dividing the MDCT spectrum coefficients into subbands to obtain a frequency bin index table corresponding to each subband; and obtaining subband energy based on the MDCT spectrum coefficients and the frequency bin index table.
[0029] Specifically, the MDCT spectrum coefficients are first divided into subbands, and the above MDCT spectrum coefficients are divided into 64 subbands, and the subband sequence number is recorded as b=0, 1, 2, ..., 63. This application takes the division of the MDCT spectrum coefficients into 64 subbands as an example. In practical applications, the subband size can be flexibly configured according to the needs to take into account the requirements of noise reduction performance and computational complexity, and then the frequency bin index table corresponding to each subband is obtained, as follows:
[0030] Index
[65] =[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,22,24,26,28,30,32,34,36,39,42,45,48,51,55,59,63, 67,71,76,81,86,92,98,105,112,119,127,135,144,154,164,175,186,198,211,225,240,256,273,291,310,330,352,375,479]
[0031] Then, the subband energy is calculated based on the MDCT spectrum coefficients and the frequency bin index table:
[0032]
[0033] After obtaining the subband energy, construct the subband energy observation matrix V noisy .
[0034] exist Figure 1 In a specific embodiment shown, the speech denoising method combining non-negative matrix decomposition and deep learning of the present application includes process S103, performing non-negative matrix decomposition on the subband energy observation matrix, and calculating the speech subband energy value and the noise subband energy value.
[0035] In a specific embodiment of the present application, non-negative matrix decomposition is performed on the subband energy observation matrix, and the speech subband energy value and the noise subband energy value are calculated, including: fixing the speech basic matrix and the noise basic matrix respectively to perform non-negative matrix decomposition on the subband energy observation matrix to obtain a speech activation matrix and a noise activation matrix; multiplying the speech basic matrix with the speech activation matrix to obtain the speech subband energy value; and multiplying the noise basic matrix with the noise activation matrix to obtain the noise subband energy value.
[0036] Specifically, the fixed speech basic matrix W speech , perform non-negative matrix decomposition on the subband energy observation matrix to obtain the speech activation matrix H′ speech .
[0037] Then, fix the noise basis matrix W noise , perform non-negative matrix factorization on the subband energy observation matrix to obtain the noise activation matrix H′ noise .
[0038] Multiply the speech basis matrix by the speech activation matrix to calculate the speech subband energy value:
[0039] V′ speech (b) = W speech H′ speech
[0040] Multiply the noise basis matrix by the noise activation matrix to calculate the noise subband energy value:
[0041] N′ noise (b) = W noise H′ noise
[0042] In a specific embodiment of the present application, the training process of the speech basic matrix and the noise basic matrix includes: selecting clean speech and noise as training sets; performing frame segmentation, windowing and discrete cosine transform on the clean speech and noise respectively to obtain the clean speech spectrum coefficients and the noise spectrum coefficients; calculating the corresponding subband energies according to the clean speech spectrum coefficients and the noise spectrum coefficients respectively, and constructing the clean speech subband energy observation matrix and the noise subband energy observation matrix accordingly; and performing non-negative matrix decomposition on the clean speech subband energy observation matrix and the noise subband energy observation matrix respectively to obtain the speech basic matrix and the noise basic matrix.
[0043] Next, the speech basic matrix W speechand the noise basis matrix W noise The training process is briefly described as follows:
[0044] (1) Selecting clean speech and noise as training sets (the following takes the training process of clean speech as an example, the training process of noise is similar and will not be repeated);
[0045] (2) performing framing, windowing and discrete cosine transform on the clean speech to obtain spectral coefficients;
[0046] (3) Calculate the subband energy and construct the observation matrix V;
[0047] (4) Perform non-negative matrix decomposition on the observation matrix V to obtain the basic matrix W (i.e., the speech basic matrix W speech ) and the activation matrix H.
[0048] For a non-negative matrix V, non-negative matrix decomposition is to find the optimal solution so that the product of W and H obtained by decomposition is an approximation of V. The steps of non-negative matrix decomposition are briefly described as follows:
[0049] (1) Construct the observation matrix V KL , there are K rows and L columns in total, each column is the amplitude spectrum of a frame, and there are L columns in total.
[0050] (2) Initialize the basis matrix W KM .
[0051] (3) Initialize the activation matrix H ML In a typical scenario, the above parameters can be K=480, M=240, and L=3~5.
[0052] (4) Iterative solution based on KL divergence criterion is briefly described as follows:
[0053]
[0054] in represents element-wise multiplication, T represents the transpose operation of the matrix, and I represents the identity matrix.
[0055] During the iteration process, the KL divergence of V and WH is calculated, and the process stops when it is less than the preset threshold. In order to avoid long calculation time caused by abnormal conditions, the number of iterations can also be preset, and the process stops when the preset number is reached, thereby obtaining the basis matrix and activation matrix.
[0056] exist Figure 1 In a specific embodiment shown, the speech denoising method combining non-negative matrix decomposition and deep learning of the present application includes process S104, calculating the first sub-band denoising gain according to the speech sub-band energy value and the noise sub-band energy value.
[0057] In this specific implementation, the first sub-band noise reduction gain is calculated:
[0058]
[0059] exist Figure 1 In a specific embodiment shown, the speech denoising method combining non-negative matrix decomposition and deep learning of the present application includes process S105, inputting the first subband denoising gain into a pre-trained deep neural network, outputting the second subband denoising gain, and performing interpolation on the second subband denoising gain to obtain all frequency bin gains.
[0060] In this specific implementation, the first sub-band noise reduction gain is input into a pre-trained deep neural network, and the pre-trained deep neural network outputs a new sub-band noise reduction gain That is, the second sub-band noise reduction gain, and then the second sub-band noise reduction gain is interpolated to obtain all frequency bin gains.
[0061] Figure 2 It is a schematic diagram of a specific example of the training process of the pre-trained deep neural network in this application.
[0062] In a specific embodiment of the present application, the training process of the pre-trained deep neural network includes: obtaining a predetermined number of clean speech and noise, and obtaining the basic matrices corresponding to the clean speech and the noise respectively; mixing the clean speech and the noise to obtain noisy speech, and using the non-negative matrix decomposition method to obtain the preliminary sub-band noise reduction gain for the noisy speech; performing discrete cosine transform on the clean speech and the noisy speech respectively to obtain the corresponding clean speech spectral coefficients and the noisy speech spectral coefficients; calculating the corresponding sub-band energies according to the clean speech spectral coefficients and the noisy speech spectral coefficients, and calculating the ideal sub-band gain; inputting the preliminary sub-band noise reduction gain into the deep neural network, training the deep neural network with the ideal sub-band gain as the target, and freezing the deep neural network when the loss level reaches the expected level to obtain the pre-trained deep neural network.
[0063] Specifically, Figure 2 As shown, first, a certain amount of pure speech and noise is obtained, and then according to the above-mentioned speech basic matrix W speech and the noise basis matrix W noise The training process obtains the speech basis matrix W corresponding to the pure speech speech The noise basis matrix W corresponding to the noise noise .
[0064] Then the clean speech is mixed with the noise to obtain the noisy speech, and the preliminary sub-band noise reduction gain is obtained using the aforementioned non-negative matrix decomposition method.
[0065] Then, feature extraction is performed on the clean speech and the noisy speech. Specifically, discrete cosine transform is performed on the clean speech and the noisy speech to output the spectral coefficients corresponding to each other, and then the sub-band energy corresponding to the clean speech and the noisy speech is calculated (the specific calculation method is the same as above, which is omitted here and will not be repeated), wherein, clean speech: x clean (n), pure speech spectrum coefficient: X clean (k), Energy of pure speech subband: subband，clean (b); Noisy speech x noisy (n), noisy speech spectrum coefficient X noisy (k), noisy speech with energy subband，noisy (b).
[0066] After obtaining the clean speech subband energy and the noisy speech subband energy through calculation, calculate the ideal subband gain:
[0067]
[0068] Then the preliminary sub-band denoising gain obtained based on non-negative matrix decomposition denoising is input into the deep neural network, and the deep neural network is trained with the ideal sub-band gain as the target, and the estimated sub-band denoising gain Gain is output. s N u N bband (b), Freezing the neural network when the loss level reaches the expected level, resulting in a pre-trained deep neural network.
[0069] Among them, this application does not restrict the selection of deep neural network. Considering the front-to-back correlation characteristics of speech frames, recurrent neural network (RNN) and the like are preferably selected.
[0070] When doing backpropagation, the loss function used is defined as:
[0071]
[0072] In a specific embodiment of the present application, interpolation is performed on the second sub-band noise reduction gain to obtain all frequency bin gains, including: performing a predetermined operation on the second sub-band noise reduction gain and a waveform symmetrical about the sub-band center frequency bin to obtain all frequency bin gains.
[0073] Specifically, all frequency bin gains are:
[0074]
[0075] Among them, Weight b (k) is a waveform symmetrical about the sub-band center frequency bin, which satisfies the following conditions:
[0076]
[0077] exist Figure 1 In a specific implementation shown, the speech denoising method combining non-negative matrix decomposition and deep learning of the present application includes process S106, obtaining denoising spectrum coefficients according to frequency bin gains and MDCT spectrum coefficients.
[0078] In a specific embodiment of the present application, the noise reduction spectrum coefficients are obtained according to the frequency bin gains and the MDCT spectrum coefficients, including: multiplying the frequency bin gains and the MDCT spectrum coefficients to obtain the noise reduction spectrum coefficients.
[0079] Specifically, the frequency bin gain is applied to the MDCT spectrum coefficients to obtain the noise reduction spectrum coefficients:
[0080]
[0081] exist Figure 1 In a specific embodiment shown, the speech denoising method combining non-negative matrix decomposition and deep learning of the present application includes process S107, performing inverse discrete cosine transform and overlap-addition on the denoised spectral coefficients, and outputting the denoised speech PCM.
[0082] Specifically,
[0083]
[0084]
[0085] Among them, w S (n) is the synthesis window, It is an overlapping and adding storage space. It is the output noise-reduced speech PCM.
[0086] Figure 3 It is a schematic diagram of a specific example of speech denoising combining non-negative matrix decomposition and deep learning in the present application.
[0087] like Figure 3As shown in a specific example, the speech noise reduction process of the Bluetooth transmitter end of the present application combining non-negative matrix decomposition and deep learning is as follows: first, input noisy speech, perform framing, windowing and discrete cosine transform to obtain MDCT spectrum coefficients, then calculate the subband energy and form a subband energy observation matrix, perform non-negative matrix decomposition on the subband energy observation matrix to obtain the activation matrix of speech and noise, then calculate the subband energy of speech and noise, thereby calculating the subband gain, and then input the subband gain into the trained deep neural network, output the new subband gain, then interpolate the new subband gain to obtain all frequency bin gains, and then apply the frequency bin gain to the MDCT spectrum coefficients to obtain the spectral coefficients after noise reduction, and then perform inverse discrete cosine transform and overlap addition on the spectral coefficients after noise reduction, and finally output the noise-reduced speech. The advantages of the present application combining non-negative matrix decomposition with deep learning to implement speech noise reduction are as follows:
[0088] (1) Generate noise reduction gain based on non-negative matrix decomposition and then correct the gain based on deep learning, which not only avoids the shortcomings of non-negative matrix decomposition, but also avoids the use of overly complex network models in deep learning;
[0089] (2) Performing non-negative matrix factorization on subbands, compared with the traditional non-negative matrix factorization based on amplitude spectrum, the size of the matrix is greatly reduced, and the amount of calculation and storage are also reduced accordingly;
[0090] (3) The deep learning model is mainly used to correct the denoising gain of non-negative matrix decomposition. Compared with the traditional end-to-end deep learning denoising model based on amplitude spectrum, the network structure is simpler, and the amount of computation and storage are reduced accordingly;
[0091] (4) Traditional noise reduction based on Fourier transform and inverse transform requires a phase spectrum, but the phase spectrum of pure speech cannot be obtained, so only the phase spectrum of noisy speech can be used. The present application is based on discrete cosine transform processing, and the phase spectrum is not required during inverse transform, thus avoiding the damage of the noisy speech phase spectrum to the reconstructed speech.
[0092] In the speech denoising method combining non-negative matrix decomposition and deep learning of the present application, the denoising gain is generated by performing non-negative matrix decomposition on the subband, and then the denoising gain is corrected based on deep learning, so as to achieve speech denoising, ensure the sound quality, avoid the shortcomings of non-negative matrix decomposition, and avoid using overly complex network models for deep learning. Based on the processing of discrete cosine transform, the phase spectrum is not required during the inverse transform. Compared with the traditional Fourier transform and inverse transform denoising which requires the phase spectrum, it avoids the damage to the reconstructed speech due to the inability to obtain the phase spectrum of pure speech and only using the phase spectrum of noisy speech; performing non-negative matrix decomposition on the subband, compared with the traditional amplitude spectrum, the size of the matrix is greatly reduced, and the amount of calculation and storage are reduced accordingly; the deep learning model is mainly used to correct the denoising gain of non-negative matrix decomposition. Compared with the traditional end-to-end deep learning denoising model based on the amplitude spectrum, the network structure is simpler, and the amount of calculation and storage are reduced accordingly.
[0093] Figure 4 It is a schematic diagram of a specific implementation of the speech noise reduction device of the present application that combines non-negative matrix decomposition and deep learning.
[0094] exist Figure 4 In a specific embodiment shown, the speech denoising device combining non-negative matrix decomposition and deep learning of the present application includes: a module 401 for inputting monophonic noisy speech PCM data and performing discrete cosine transform to obtain MDCT spectrum coefficients; a module 402 for calculating subband energy according to MDCT spectrum coefficients and constructing a subband energy observation matrix; a module 403 for performing non-negative matrix decomposition on the subband energy observation matrix and calculating speech subband energy values and noise subband energy values; a module 404 for calculating the first subband denoising gain according to the speech subband energy values and the noise subband energy values; a module 405 for inputting the first subband denoising gain into a pre-trained deep neural network, outputting the second subband denoising gain, and performing interpolation on the second subband denoising gain to obtain all frequency bin gains; a module 406 for obtaining denoising spectrum coefficients according to the frequency bin gains and MDCT spectrum coefficients; a module 407 for performing inverse discrete cosine transform and overlap-addition on the denoising spectrum coefficients to output denoised speech PCM.
[0095] In a specific embodiment of the present application, monophonic noisy speech PCM data is input and discrete cosine transform is performed to obtain MDCT spectrum coefficients, including: framing the monophonic noisy speech PCM data with a fixed frame length and windowing the obtained audio frames; and discrete cosine transform is performed on the windowed audio frames to obtain MDCT spectrum coefficients.
[0096] In a specific embodiment of the present application, subband energy is calculated based on MDCT spectrum coefficients, including: dividing the MDCT spectrum coefficients into subbands to obtain a frequency bin index table corresponding to each subband; and obtaining subband energy based on the MDCT spectrum coefficients and the frequency bin index table.
[0097] In a specific embodiment of the present application, non-negative matrix decomposition is performed on the subband energy observation matrix, and the speech subband energy value and the noise subband energy value are calculated, including: fixing the speech basic matrix and the noise basic matrix respectively to perform non-negative matrix decomposition on the subband energy observation matrix to obtain a speech activation matrix and a noise activation matrix; multiplying the speech basic matrix with the speech activation matrix to obtain the speech subband energy value; and multiplying the noise basic matrix with the noise activation matrix to obtain the noise subband energy value.
[0098] In a specific embodiment of the present application, the training process of the speech basic matrix and the noise basic matrix includes: selecting clean speech and noise as training sets; performing frame segmentation, windowing and discrete cosine transform on the clean speech and noise respectively to obtain the clean speech spectrum coefficients and the noise spectrum coefficients; calculating the corresponding subband energies according to the clean speech spectrum coefficients and the noise spectrum coefficients respectively, and constructing the clean speech subband energy observation matrix and the noise subband energy observation matrix accordingly; and performing non-negative matrix decomposition on the clean speech subband energy observation matrix and the noise subband energy observation matrix respectively to obtain the speech basic matrix and the noise basic matrix.
[0099] In a specific embodiment of the present application, the training process of the pre-trained deep neural network includes: obtaining a predetermined number of clean speech and noise, and obtaining the basic matrices corresponding to the clean speech and the noise respectively; mixing the clean speech and the noise to obtain noisy speech, and using the non-negative matrix decomposition method to obtain the preliminary sub-band noise reduction gain for the noisy speech; performing discrete cosine transform on the clean speech and the noisy speech respectively to obtain the corresponding clean speech spectral coefficients and the noisy speech spectral coefficients; calculating the corresponding sub-band energies according to the clean speech spectral coefficients and the noisy speech spectral coefficients, and calculating the ideal sub-band gain; inputting the preliminary sub-band noise reduction gain into the deep neural network, training the deep neural network with the ideal sub-band gain as the target, and freezing the deep neural network when the loss level reaches the expected level to obtain the pre-trained deep neural network.
[0100] In a specific embodiment of the present application, interpolation is performed on the second sub-band noise reduction gain to obtain all frequency bin gains, including: performing a predetermined operation on the second sub-band noise reduction gain and a waveform symmetrical about the sub-band center frequency bin to obtain all frequency bin gains.
[0101] In the speech denoising device combining non-negative matrix decomposition and deep learning of the present application, the denoising gain is generated by performing non-negative matrix decomposition on the subband, and then the denoising gain is corrected based on deep learning, so as to achieve speech denoising, ensure the sound quality, avoid the shortcomings of non-negative matrix decomposition, and avoid using overly complex network models for deep learning. Based on the processing of discrete cosine transform, the phase spectrum is not required during the inverse transform. Compared with the traditional Fourier transform and inverse transform denoising which requires the phase spectrum, it avoids the damage to the reconstructed speech due to the inability to obtain the phase spectrum of pure speech and only using the phase spectrum of noisy speech; performing non-negative matrix decomposition on the subband, compared with the traditional amplitude spectrum, the size of the matrix is greatly reduced, and the amount of calculation and storage are reduced accordingly; the deep learning model is mainly used to correct the denoising gain of non-negative matrix decomposition. Compared with the traditional end-to-end deep learning denoising model based on the amplitude spectrum, the network structure is simpler, and the amount of calculation and storage are reduced accordingly.
[0102] In a specific embodiment of the present application, a computer-readable storage medium stores computer instructions, which are operated to execute the speech denoising method combining non-negative matrix decomposition and deep learning described in any embodiment. The storage medium can be directly in hardware, in a software module executed by a processor, or in a combination of the two.
[0103] The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium.
[0104] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In an alternative, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.
[0105] In a specific embodiment of the present application, a computer device includes a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the speech denoising method combining non-negative matrix decomposition and deep learning described in any embodiment.
[0106] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0108] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structural transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A speech denoising method combining non-negative matrix decomposition and deep learning, characterized in that: include: Input mono noisy speech PCM data and perform discrete cosine transform to obtain MDCT spectrum coefficients; Calculating subband energy according to the MDCT spectrum coefficients and constructing a subband energy observation matrix; Performing non-negative matrix decomposition on the sub-band energy observation matrix, and calculating speech sub-band energy values and noise sub-band energy values; Calculating a first sub-band noise reduction gain according to the speech sub-band energy value and the noise sub-band energy value; Inputting the first sub-band noise reduction gain into a pre-trained deep neural network, outputting a second sub-band noise reduction gain, and performing interpolation on the second sub-band noise reduction gain to obtain all frequency bin gains; Obtaining noise reduction spectrum coefficients according to the frequency bin gain and the MDCT spectrum coefficients; as well as Inverse discrete cosine transform and overlap-add are performed on the noise reduction spectrum coefficients to output noise reduction speech PCM.
2. The method for speech denoising combining non-negative matrix decomposition and deep learning according to claim 1, characterized in that: The input mono-channel noisy speech PCM data is subjected to discrete cosine transform to obtain MDCT spectrum coefficients, including: Frame the mono-channel noisy speech PCM data with a fixed frame length, and perform windowing on the obtained audio frames; and A discrete cosine transform is performed on the windowed audio frame to obtain the MDCT spectrum coefficient.
3. The speech denoising method combining non-negative matrix decomposition and deep learning according to claim 1, characterized in that: The calculating sub-band energy according to the MDCT spectrum coefficients comprises: Dividing the MDCT spectrum coefficients into sub-bands to obtain a frequency bin index table corresponding to each of the sub-bands; and The subband energy is obtained according to the MDCT spectrum coefficients and the frequency bin index table.
4. The method for speech denoising combining non-negative matrix decomposition and deep learning according to claim 1, characterized in that: The performing non-negative matrix decomposition on the sub-band energy observation matrix and calculating the speech sub-band energy value and the noise sub-band energy value comprises: The speech basic matrix and the noise basic matrix are fixed respectively to perform non-negative matrix decomposition on the sub-band energy observation matrix to obtain a speech activation matrix and a noise activation matrix; Multiplying the speech basic matrix by the speech activation matrix to obtain the speech sub-band energy value; and The noise basic matrix is multiplied by the noise activation matrix to obtain the noise subband energy value.
5. The method for speech denoising combining non-negative matrix decomposition and deep learning according to claim 3, characterized in that: The interpolating the second sub-band noise reduction gain to obtain all frequency bin gains includes: The second sub-band noise reduction gain is subjected to a predetermined operation with a waveform symmetrical with respect to the sub-band center frequency bin to obtain all the frequency bin gains.
6. The speech denoising method combining non-negative matrix decomposition and deep learning according to claim 1, characterized in that: The training process of the speech basic matrix and the noise basic matrix includes: Select pure speech and noise as training sets; Performing framing, windowing and discrete cosine transform on the clean speech and the noise respectively to obtain clean speech spectrum coefficients and noise spectrum coefficients; Calculating corresponding subband energies according to the clean speech spectrum coefficients and the noise spectrum coefficients respectively, and constructing a clean speech subband energy observation matrix and a noise subband energy observation matrix accordingly; and Non-negative matrix decomposition is performed on the clean speech subband energy observation matrix and the noise subband energy observation matrix respectively to obtain the speech basic matrix and the noise basic matrix.
7. The speech denoising method combining non-negative matrix decomposition and deep learning according to claim 1, characterized in that: The training process of the pre-trained deep neural network includes: Acquire a predetermined number of clean speech and noise, and respectively acquire basic matrices corresponding to the clean speech and the noise; The clean speech and the noise are mixed to obtain a noisy speech, and a preliminary sub-band noise reduction gain is obtained by using a non-negative matrix decomposition method on the noisy speech; Performing discrete cosine transform on the clean speech and the noisy speech respectively to obtain corresponding clean speech spectrum coefficients and noisy speech spectrum coefficients; Calculating the sub-band energies corresponding to the clean speech spectrum coefficients and the noisy speech spectrum coefficients respectively, and calculating the ideal sub-band gains; The preliminary sub-band noise reduction gain is input into a deep neural network, the deep neural network is trained with the ideal sub-band gain as a target, and when the loss level reaches an expected level, the deep neural network is frozen to obtain the pre-trained deep neural network.
8. A speech noise reduction device combining non-negative matrix decomposition and deep learning, characterized in that: include: A module for inputting monophonic noisy speech PCM data and performing discrete cosine transform to obtain MDCT spectrum coefficients; A module for calculating sub-band energy according to the MDCT spectrum coefficients and constructing a sub-band energy observation matrix; A module for performing non-negative matrix decomposition on the sub-band energy observation matrix and calculating a speech sub-band energy value and a noise sub-band energy value; A module for calculating a first sub-band noise reduction gain according to the speech sub-band energy value and the noise sub-band energy value; A module for inputting the first sub-band noise reduction gain into a pre-trained deep neural network, outputting a second sub-band noise reduction gain, and performing interpolation on the second sub-band noise reduction gain to obtain all frequency bin gains; A module for obtaining noise reduction spectrum coefficients according to the frequency bin gains and the MDCT spectrum coefficients; A module for performing inverse discrete cosine transform and overlap-addition on the noise reduction spectrum coefficients to output the noise reduction speech PCM.
9. A computer-readable storage medium storing computer instructions, wherein the computer instructions are operated to execute the speech denoising method combining non-negative matrix decomposition and deep learning as described in any one of claims 1-7.
10. A computer device comprising a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the speech denoising method combining non-negative matrix decomposition and deep learning as described in any one of claims 1-7.