A comfort noise generation method based on time-frequency masking estimation and its application

Abstract

The present invention relates to the technical field of noise processing, and specifically discloses a method for generating comfortable noise based on time-frequency masking estimation and its application, wherein the method comprises the following steps: S1, converting the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtaining the spectrum X(l,k) of the microphone signal in the lth frame and the kth frequency band; S2, estimating the power spectrum density of the comfort noise; S3, generating comfort noise of corresponding energy; S4, synthesizing the target speech. On the one hand, the stationary noise component is estimated based on the time-frequency masking information obtained based on deep learning adopted by the scheme, which can avoid the speech energy being accumulated to the stationary noise component, resulting in excessive comfort noise; on the other hand, the time-frequency unit is selectively increased to cause comfort, so as to avoid the speech-dominated time-frequency unit from introducing noise.

Description

Technical Field

[0001] The present invention relates to the technical field of noise processing, and in particular to a method for generating comfortable noise based on time-frequency masking estimation and its application. Background Art

[0002] Noise suppression and speech enhancement have always been a key technology to improve the quality of voice communication in conference systems or conference equipment. The traditional signal processing method is to track the noise power spectrum density and speech power spectrum density in the signal, and then construct a masking value from 0 to 1 in the frequency domain based on Wiener filtering. After masking the microphone signal, the purpose of suppressing background noise is achieved. The disadvantage of signal processing technology is that it cannot effectively handle non-stationary noise in the environment, and there is also the disadvantage of excessive speech distortion under strong noise interference. At present, time-frequency masking information estimation based on deep learning is another common method for noise suppression. The main idea is to estimate the time-frequency masking value directly from the mixed signal by training the noisy data set to the pure speech signal. The deep learning method can better handle non-stationary noise, but there is also the problem of distortion caused by speech over-cancellation.

[0003] Therefore, in summary, the main shortcomings of the prior art are:

[0004] The method of using signal processing to track the stationary components in the background noise has the disadvantage of excessive comfort noise in scenes with large ambient noise.

[0005] The existing method of generating comfort noise is to add noise to all time-frequency units without any differentiation, which will result in a certain amount of noise being added to the speech-dominated time-frequency area.

[0006] How to significantly improve the quality of auditory perception within a moderate comfortable noise range is a difficult problem that needs to be solved urgently.

[0007] The existing solution is to estimate the stationary components in the ambient noise and then generate white noise of equal energy and add it to the spectrum to reduce the impact of speech distortion on auditory perception.

[0008] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0009] The object of the present invention is to provide a comfort noise generation method based on time-frequency masking estimation and its application, which can improve the communication quality, noise suppression and speech enhancement in applications such as voice conferencing systems.

[0010] To achieve the above object, the present invention provides a method for generating comfort noise based on time-frequency masking estimation, comprising the following steps:

[0011] S1, convert the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtain the spectrum X(l,k) of the microphone signal in the lth frame and the kth frequency band;

[0012] S2, estimated comfort noise power spectral density;

[0013] S3, generates comfort noise of corresponding energy;

[0014] S4, synthesize target speech.

[0015] In a specific implementation scenario, the first step is to decompose the microphone array signal. The Fourier transform is used to convert the suitable signal into a frequency domain signal to obtain a spectrum, which is convenient for subsequent noise processing. The second step is to estimate the power spectrum density of comfortable noise for the frequency domain signal. This step includes three steps in sequence: noise power spectrum density estimation, stationary noise power spectrum density estimation, and comfort noise energy estimation. Among them, the video masking information in the prior art is used in the noise power spectrum density estimation. The third step is to generate a comfortable noise spectrum. It plays a connecting role and facilitates the subsequent further processing and analysis of the speech. The fourth step is target speech estimation. This step includes the target speech frequency domain estimation. After estimation, the inverse Fourier transform is performed to obtain the target time domain signal, that is, the target speech signal, which can be output.

[0016] Optionally, the calculation formula of the spectrum X(l,k) in S1 is as follows:

[0017]

[0018] Wherein, N is the frame length of 512, w(n) is the Hamming window of frame length of 512, n is the time label, l is the time frame number, k is the frequency number, and X(l,k) is the spectrum of the microphone signal in the lth frame and the kth frequency band.

[0019] Optionally, the S2 specifically includes:

[0020] S21, obtain the time-frequency masking value M(l, k), and for each frequency band k, calculate the ambient noise power spectrum density ρ v (k):

[0021]

[0022] Where || represents the modulus of the complex number, α is the smoothing factor between adjacent frames, and its value range is between 0 and 1;

[0023] S22, for each frequency band k, estimate the stationary noise power spectral density ρ min (k):

[0024]

[0025] The stationary noise power spectrum density represents the minimum component in the tracked noise, that is, the minimum value of the noise component in the signal, α is the smoothing factor and has the same value as in step S21, and γ represents the stationary noise control factor and has a value less than 1;

[0026] S23, calculate the comfort noise energy ζ:

[0027]

[0028] The value of K is equal to half of the frame length.

[0029] Optionally, the smoothing factor α=0.95.

[0030] Optionally, the stationary noise control factor γ=0.08.

[0031] Optionally, the S3 specifically includes:

[0032] Generate comfort noise power spectrum υ(l,k):

[0033]

[0034] Here, σ(n) is a white noise sequence with energy 1 and length 512.

[0035] Optionally, the S4 specifically includes:

[0036] S41, calculating the frequency domain estimation of the target speech according to the following formula:

[0037]

[0038] Among them, spectrum X(l,k), time-frequency masking value M(l,k), comfort noise power spectrum υ(l,k);

[0039] S42, perform inverse Fourier transform to obtain the target speech time domain estimation:

[0040]

[0041] Where w(k) is a Hamming window with a frame length of 512.

[0042] The present invention also provides a comfort noise generation system based on time-frequency masking estimation, wherein the system is used to implement a comfort noise generation method based on time-frequency masking estimation, comprising:

[0043] The signal decomposition module is used to convert the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtain the spectrum X(l,k) of the microphone signal in the lth frame and the kth frequency band;

[0044] A comfort noise power spectrum density estimation module, used for estimating the comfort noise power spectrum density;

[0045] A comfort noise generating module, used for generating comfort noise of corresponding energy;

[0046] The target speech synthesis module is used to synthesize the target speech.

[0047] Optionally, the comfort noise power spectrum density estimation module includes a noise power spectrum density estimation module, a stationary noise power spectrum density estimation module and a comfort noise energy estimation module that sequentially processes the signal.

[0048] The present invention also provides an electronic device, comprising a memory and a processor, wherein the processor is used to implement the steps of a method for generating comfortable noise based on time-frequency masking estimation when executing a computer management program stored in the memory.

[0049] Compared with the prior art, the method for generating comfort noise based on time-frequency masking estimation and its application provided by the present invention include the following steps: S1, converting the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtaining the spectrum X(l,k) of the microphone signal in the lth frame and the kth frequency band; S2, estimating the power spectrum density of comfort noise; S3, generating comfort noise of corresponding energy; S4, synthesizing the target speech. On the one hand, the stationary noise component is estimated based on the time-frequency masking information obtained by deep learning adopted by the scheme, which can avoid the speech energy being accumulated to the stationary noise component, resulting in excessive comfort noise; on the other hand, the time-frequency unit is selectively increased to cause comfort, so as to avoid the introduction of noise into the speech-dominated time-frequency unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a functional principle block diagram of a comfort noise generation system based on time-frequency masking estimation according to the present invention. DETAILED DESCRIPTION

[0051] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.

[0052] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0053] Embodiment 1

[0054] A method for generating comfort noise based on time-frequency masking estimation according to a preferred embodiment of the present invention comprises the following steps:

[0055] S1, convert the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtain the spectrum X(l,k) of the microphone signal in the lth frame and the kth frequency band;

[0056] S2, estimated comfort noise power spectral density;

[0057] S3, generates comfort noise of corresponding energy;

[0058] S4, synthesize target speech.

[0059] In a preferred solution, the spectrum X(l,k) in S1 is calculated as follows:

[0060] The time-frequency domain expression is obtained by performing short-time Fourier transform on the time domain signal x(n):

[0061]

[0062] Where N is the frame length 512, w(n) is the Hamming window of length 512, n is the time tag, l is the time frame number, and k is the frequency number. X(l,k) is the spectrum of the microphone signal in the kth frequency band in the lth frame. The frequency band refers to the signal component corresponding to a certain frequency. Based on the Hamming window function, the Hamming window value corresponding to each sample time point n can be obtained. The Hamming window is a fixed existing technology and will not be described in detail here.

[0063] Embodiment 2

[0064] This embodiment is the same as the comfort noise generation method based on time-frequency masking estimation in Embodiment 1, and the differences are as follows:

[0065] S2 specifically includes: Assuming that the time-frequency masking value estimated by deep learning is M(l,k), for each frequency band k, calculate the power spectrum density ρ of the ambient noise v (k):

[0066]

[0067] Where || represents the modulus of the complex number, α is the smoothing factor between adjacent frames, and its value range is between 0 and 1. The present invention preferably sets α=0.95. If the value is too small, the power spectral density estimation will change too much and there will be an instability defect. If the value is too high, the energy estimation will be too stable and the ability to model non-stationary noise will be reduced. The prior art M(l,k) is adopted, which is a masking value obtained by deep learning estimation, and its value range is between 0 and 1. When M(l,k) is less than 0.5, it means that the time-frequency unit is dominated by environmental noise, and the noise power spectral density can be updated. Otherwise, it means that the time-frequency unit is dominated by speech, and the updating of the noise power spectral density is stopped. The result of this step is used to update the stationary noise power spectral density as described below (3).

[0068] For each frequency band k, estimate the stationary noise power spectral density ρ min (k):

[0069]

[0070] The power spectrum density represents the minimum component in the tracked noise, that is, the minimum value of the noise component in the signal, without directly smoothing the microphone signal. Among them, α is the smoothing factor, which is the same as in step 2. γ represents the stationary noise control factor. Since the stationary noise is part of the noise, the stationary noise energy is less than the noise energy, that is, γ < 1. The control factor of the present invention is γ = 0.08. This value can generate moderate comfort noise energy and avoid excessive comfort noise. This step is used to calculate the comfort noise energy.

[0071] Calculate the comfort noise energy ζ:

[0072]

[0073] The physical meaning of this step is to calculate the value ρ for all frequency bands k. min (k) Calculate the average, where K represents the number of all frequency bands. The value of K is equal to half of the frame length, that is, 256. The comfort noise energy in this step is used for the subsequent steps.

[0074] Step S3 specifically includes: generating a comfort noise power spectrum υ(l,k):

[0075]

[0076] Among them, σ(n) is a white noise sequence with energy 1 and length 512. This step uses the following step to calculate the final speech spectrum.

[0077] Step S4 specifically includes: obtaining a frequency domain estimation of the target speech according to the comfort noise power spectrum obtained in step (5):

[0078]

[0079] That is, when the energy after time-frequency masking is less than the energy of comfort noise, comfort noise is added to the time-frequency unit to avoid distortion caused by local energy loss. At the same time, if the energy after time-frequency masking is greater than the energy of comfort noise, no noise is added to avoid excessive increase in noise energy.

[0080] Perform inverse Fourier transform to obtain the target speech time domain estimate:

[0081]

[0082]

[0083] Where w(k) is a Hamming window with a frame length of 512.

[0084] The time domain estimated signal can be directly converted into a voltage signal through digital-to-analog conversion, and the enhanced speech can be played by the speaker.

[0085] like Figure 1 As shown, an embodiment of the present invention further provides a comfort noise generation system based on time-frequency masking estimation, wherein the system is used to implement the comfort noise generation method based on time-frequency masking estimation as described in the first two embodiments, wherein the system comprises:

[0086] The signal decomposition module is used to convert the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtain the spectrum X(l,k) of the microphone signal in the lth frame and the kth frequency band;

[0087] A comfort noise power spectrum density estimation module, used for estimating the comfort noise power spectrum density;

[0088] A comfort noise generating module, used for generating comfort noise of corresponding energy;

[0089] The target speech synthesis module is used to synthesize the target speech.

[0090] Specifically, the comfort noise power spectrum density estimation module includes a noise power spectrum density estimation module, a stationary noise power spectrum density estimation module and a comfort noise energy estimation module that sequentially processes the signal.

[0091] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the processor is used to implement the steps of the comfort noise generation method based on time-frequency masking estimation as described in the first two embodiments when executing a computer management program stored in the memory.

[0092] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A comfort noise generation method based on time-frequency masking estimation, characterized in that: The following steps are involved: S1, convert the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtain the spectrum X(l, k) of the microphone signal in the first frame and the kth frequency band; S2, estimated comfort noise power spectral density; S3, generates comfort noise of corresponding energy; S4, synthetic target speech; Wherein, the S2 specifically includes: S21, obtain the time-frequency masking value M(1, k), and for each frequency band k, calculate the ambient noise power spectrum density ρ v (k): Where || represents the modulus of the complex number, α is the smoothing factor between adjacent frames, and its value range is between 0 and 1; S22, for each frequency band k, estimate the stationary noise power spectral density ρ min (k): The stationary noise power spectrum density represents the minimum component in the tracked noise, that is, the minimum value of the noise component in the signal, α is the smoothing factor and has the same value as in step S21, and γ represents the stationary noise control factor and has a value less than 1; S23, calculate the comfort noise energy ζ: The value of K is equal to half of the frame length.

2. The comfort noise generation method based on time-frequency masking estimation according to claim 1, characterized in that: The calculation formula of the spectrum X(l, k) in S1 is as follows: Wherein, N is the frame length of 512, w(n) is the Hamming window of frame length 512, n is the time label, 11 is the time frame number, k is the frequency number, X(l, k) is the spectrum of the microphone signal in the first frame and the kth frequency band, x(n) is the time domain signal picked up by the microphone array element, and j is the imaginary unit.

3. The comfort noise generation method based on time-frequency masking estimation according to claim 1, characterized in that: The smoothing factor α == 0.

95.

4. The comfort noise generation method based on time-frequency masking estimation according to claim 1, characterized in that: The stationary noise control factor γ==0.

08.

5. The comfort noise generation method based on time-frequency masking estimation according to claim 2, characterized in that: The S3 specifically includes: Generate comfort noise power spectrum υ(l, k): Here, σ(n) is a white noise sequence with energy 1 and length 512.

6. The comfort noise generation method based on time-frequency masking estimation according to claim 5, characterized in that: The S4 specifically includes: S41, calculating the frequency domain estimation of the target speech according to the following formula: Among them, spectrum X(l, k), time-frequency masking value M(1, k), comfort noise power spectrum υ(l, k); S42, perform inverse Fourier transform to obtain the target speech time domain estimation: Where w(k) is a Hamming window with a frame length of 512.

7. A comfort noise generation system based on time-frequency masking estimation, characterized in that: The system is used to implement the comfort noise generation method based on time-frequency masking estimation according to any one of claims 1 to 6, comprising: The signal decomposition module is used to convert the time domain signal x(n) picked up by the microphone array element into a time-frequency domain signal, and obtain the spectrum X(l, k) of the microphone signal in the first frame and the kth frequency band; A comfort noise power spectrum density estimation module, used for estimating the comfort noise power spectrum density; A comfort noise generating module, used for generating comfort noise of corresponding energy; The target speech synthesis module is used to synthesize the target speech.

8. The comfort noise generation system based on time-frequency masking estimation according to claim 7, characterized in that: The comfort noise power spectrum density estimation module comprises a noise power spectrum density estimation module, a stationary noise power spectrum density estimation module and a comfort noise energy estimation module which process the signal in sequence.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the processor is used to implement the steps of the comfort noise generation method based on time-frequency masking estimation as described in any one of claims 1 to 6 when executing a computer management program stored in the memory.

Images