Simultaneous noise reduction and dereverberation via low-latency deep learning

Abstract

The present invention proposes a method, comprising extracting spectral features of a cochlear impulse response of an auditory system from a frame of an audio sample; obtaining an estimate of a speech signal in the frame from a neural network (the neural network has been trained in this manner during a training phase: that is, it accepts spectral features as input and outputs an estimate of a speech signal), wherein the estimate of the speech signal includes reverberation but excludes noise present in the frame; mapping the estimate of the speech signal to the frequency domain of the frame using mapping parameters obtained during the training phase, thereby obtaining an estimate of the reverberant speech spectrum in the frame; and obtaining a dereverberated frame of the frame from a time-distributed neural network, wherein the estimate of the reverberant speech spectrum in the frame is used as input to the time-distributed neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. patent application Ser. No. 17 / 072,605, filed on October 16, 2020, entitled “Simultaneous Noise Reduction and De-Reverberation via Low-Latency Deep Learning,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates generally to the field of speech enhancement, and more particularly to simultaneous dereverberation and noise reduction techniques in real-time applications. Background Art

[0004] Interactive communication often occurs online across various communication channels using different media types. Examples include real-time communication using video conferencing, video streaming, or simple phone calls. Video can contain both audio and video content. A user (the sender) can send user-generated content (such as a video) to one or more recipients. For example, a concert can be live-streamed to a large audience. Another example is a teacher live-streaming a class to students. Another example is a group of users engaging in a real-time chat that includes live video.

[0005] Audio signals (such as speech) in such real-time communications can be corrupted by reverberation and background noise. Reverberation occurs when sound or signals are absorbed by objects (such as walls and furniture) in a space (such as a room or hall), resulting from multiple reflections and the resulting attenuation. Background noise can come from other interfering sound sources. Both reverberation and background noise can degrade the quality and clarity of audio (such as speech). Summary of the Invention

[0006] In one aspect, the present invention provides a method for denoising and dereverberation of an audio sample. The method comprises extracting spectral features of the cochlear impulse response of the auditory system from a frame of the audio sample; obtaining an estimate of the speech signal in the frame from a neural network (the neural network has been trained in a training phase to accept spectral features as input and output an estimate of the speech signal), wherein the estimate of the speech signal includes reverberation but excludes noise present in the frame; mapping the estimate of the speech signal to the frequency domain of the frame using mapping parameters obtained in the training phase, thereby obtaining an estimate of the reverberant speech spectrum in the frame; and obtaining a dereverberated frame of the frame from a time-distributed neural network, wherein the estimate of the reverberant speech spectrum in the frame is used as input to the time-distributed neural network.

[0007] In a second aspect, the present invention provides a system for denoising and dereverberation of audio samples. The system includes a denoising module, an adaptive mapping module, and a dereverberation module. The denoising module is configured to extract spectral features related to the cochlear impulse response of the auditory system from a frame of audio samples; obtain an estimate of the speech signal in the frame from a neural network (the neural network has been trained in such a manner during a training phase: that is, it accepts spectral features as input and outputs an estimate of the speech signal), wherein the estimate of the speech signal includes reverberation but excludes noise present in the frame. The adaptive mapping module is configured to use the mapping parameters obtained during the training phase to map the estimate of the speech signal to the frequency domain of the frame, thereby obtaining an estimate of the reverberant speech spectrum in the frame. The dereverberation module is configured to obtain a dereverberated frame of the frame from a time-distributed neural network, wherein the estimate of the reverberant speech spectrum in the frame is used as input to the time-distributed neural network.

[0008] In a third aspect, the present invention provides an implementation device including a processor. The processor is configured to extract spectral features related to the cochlear impulse response of the auditory system from a frame of audio samples; obtain an estimate of the speech signal in the frame from a neural network (the neural network has been trained in such a manner during a training phase: that is, it accepts spectral features as input and outputs an estimate of the speech signal), wherein the estimate of the speech signal includes reverberation but excludes noise present in the frame; use the mapping parameters obtained during the training phase to map the estimate of the speech signal to the frequency domain of the frame, thereby obtaining an estimate of the reverberant speech spectrum in the frame; and obtain a dereverberated frame of the frame from a time-distributed neural network, wherein the estimate of the reverberant speech spectrum in the frame is used as input to the time-distributed neural network.

[0009] Each of the above aspects can be implemented using a variety of different implementations. For example, each of the above aspects can be implemented using a suitable computer program, which can be implemented on a suitable carrier medium, which can be a tangible carrier medium (such as a disk) or an intangible carrier medium (such as a communication signal). Suitable devices can also be used to implement various aspects of the functions, which can take the form of a programmable computer running a computer program, which is configured to implement the methods and / or techniques described in the present invention. The above aspects can also be used in combination so that the functions described in one aspect of the technology can be implemented in another aspect of the technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The description herein refers to the drawings, wherein like numbers refer to like components throughout the various views.

[0011] Figure 1 3 is an example diagram of a training phase technology for performing noise reduction and dereverberation on real-time audio, drawn according to an embodiment of the present invention.

[0012] Figure 2 FIG. 4 is a schematic diagram of mapping sub-band gains to frequency bins according to an embodiment of the present invention.

[0013] Figure 3 is an example diagram of a technology for denoising and dereverberation of real-time audio, drawn according to an embodiment of the present invention.

[0014] Figure 4 FIG. 1 is a block diagram of an example computing device according to an embodiment of the present invention.

[0015] Figure 5 is an example of the structure of a deep feature convolutional neural network drawn according to an embodiment of the present invention.

[0016] Figure 6 3 is a flowchart illustrating a technique for performing noise reduction and dereverberation on audio samples according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] As mentioned above, audio signals (such as speech) may be corrupted by reverberation and / or background noise. Various noise reduction and dereverberation techniques have been developed to date, but these techniques may require excessive time or computing resources to execute, resulting in unnecessary delays and latency, especially in real-time communications (such as real-time video communications).

[0018] The deep learning technology proposed in accordance with the present invention can simultaneously and effectively eliminate the blurring effect caused by reverberation and effectively suppress ambient noise in the audio signal. Even when transient noise or room impulses are constantly changing, it is possible to simultaneously and effectively eliminate (or at least reduce) the effects of transient noise (i.e., background noise) and offset the effects of the room impulse response (i.e., reverberation). The deep learning technology described herein can be used to implement low-latency dereverberation and noise reduction algorithms in real-time applications (such as video conferencing). Such technology can be applied to user devices with limited computing resources, such as smartphones and handheld devices.

[0019] The technology described herein can be used at the sender user's device (i.e., the sending device), or at the recipient user's device (e.g., the receiving device), or both. In this description, the sender user refers to a person who may be speaking and whose voice will be sent to and heard by the recipient user. The sender user's voice may be corrupted by background noise and / or reverberation. A central server (e.g., a cloud-based server) can also use the technology described herein, which can receive audio signals from the sender user and transmit the audio signals to the recipient user.

[0020] Machine learning (ML) is used to reduce noise and dereverberation (i.e., remove reverberation artifacts) from audio signals (e.g., speech signals). The present invention can be thought of as consisting of two parts: the first part is for background noise reduction, and the second part is for dereverberation. However, embodiments of the present invention perform both noise reduction and dereverberation functions simultaneously, as explained below.

[0021] As we all know, machine learning involves a training phase (e.g., staged training) and an inference phase (e.g., classification, usage, production, etc.). Typically, the machine learning models trained during the training phase are directly used in the inference phase. However, to reduce complexity (as described below), in real-time applications such as noise and reverberation reduction (e.g., audio or video conferencing), although several ML models are trained during the training phase, not all of them are used during the inference phase.

[0022] Figure 1 FIG2 is an exemplary diagram illustrating a training phase technique 100 for performing noise reduction and dereverberation on real-time audio according to an embodiment of the present invention. Technique 100 receives training data. The training data includes noise and reverberation data samples (e.g., noise and reverberation speech samples, also referred to herein as mixed speech) and corresponding reference samples (e.g., clean speech without reverberation).

[0023] The technique 100 includes a feature extraction stage (including modules 116 and 118), a noise reduction stage 102, an adaptive mapping stage 104, and a dereverberation stage 106. For a given training speech mixture, the technique 100 can divide the training speech mixture into a plurality of frames. The duration of each frame can be 10 milliseconds, 50 milliseconds, or shorter or longer. The technique 100 then processes at least a portion of the frames (e.g., each frame).

[0024] Without loss of generality, an implementation method is used as an example: Technology 100 can be summarized as performing the following steps (such as stages, sub-steps, etc.): 1) extracting gammatone auditory features from the mixed speech (i.e., from the audio frame) and extracting the fast Fourier transform (FFT) spectrum from the corresponding reverberation-free clean speech (i.e., the corresponding audio frame therein); 2) training two neural networks in parallel to separate speech and noise in units of sub-bands divided by equivalent rectangular bandwidth (ERB); 3) adjusting the adaptive parameters of the gain mapping according to the FFT spectrum of the mixed speech; and 4) training a time distributed deep neural network (TD-DNN) to remove late reverberation.

[0025] We illustrate this by applying technique 100 to one frame of training data (referred to herein as the "current frame"), but note that the same operation can be applied to every frame of data.

[0026] Module 116 can be used to extract short-term spectral features of the sound waveform from the current frame. The spectral features can refer to the cochlear impulse response of the auditory system. In one example, a bandpass filter can be used to extract short-term spectral features from the current frame. In another example, a gammatone filter bank can also be used. In yet another example, a 64-channel filter bank can also be used. Therefore, 64 channels of gammatone features can be extracted from the current frame. The gammatone filter bank represents the cochlear impulse response of the auditory system. In other words, the gammatone filter bank can approximate the auditory response of nerve fibers. The impulse response of the gammatone filter is shown in formula (1):

[0027]

[0028] In formula (1), A represents the amplitude of the audio signal of the frame; n represents the filter order, which largely determines the slope of the filter skirt; b i (in Hertz) represents the bandwidth and controls the duration of the impulse response function; f i (in Hertz) represents the center frequency of the carrier, (in radians) represents the phase of the carrier. Bandwidth b i Can be b i =1.019ERB(f i ), where ERB represents the equivalent rectangular bandwidth. The impulse response of the gammatone filter bank can provide suitable data for the movement of the basilar membrane in the cochlea.

[0029] As mentioned above, by having different center frequencies f i The 64-channel gammatone filter bank can filter the current frame to obtain the corresponding feature data. This allows the acquisition of 64 features for the current frame. A gammatone filter bank is typically defined such that the filter center frequency is proportional to the bandwidth across the entire frequency range, a concept known as equivalent rectangular bandwidth (ERB) scaling.

[0030] The features extracted from the gammatone filter bank can be used as input data for the noise reduction stage 102. The noise reduction module in the training stage (i.e., in technology 100) separates the additive noise in the speech by estimating the ERB gain for each subband (i.e., each frequency point) of the current frame. The ERB scale is a nonlinear frequency scale that can be based on the human auditory system. The ERB scale presents a narrower resolution at low frequencies and a coarser resolution at high frequencies. Therefore, the use of the ERB gain can effectively preserve the harmonics of the speech and suppress the noise in the mixed speech. In addition, compared with the short-time Fourier transform (STFT) time-frequency representation, the use of gammatone features and ERB gains can greatly reduce the computational complexity and the number of nodes required compared to a neural network using STFT features as input. For example, because the STFT has more features than the gammatone filter, the amount of computation required to calculate the first layer of the network in the input will be greater.

[0031] The denoising stage 102 includes a first deep feature convolutional neural network (i.e., FCN1 108) and a second deep feature convolutional neural network (i.e., FCN2 112). Figure 5 The structures of FCN1 108 and FCN2 112 are described respectively. FCN1 108 and FCN2 112 are trained together using a mean squared error loss function between the predicted speech-only gammatone features and the speech-only gammatone features of the reference samples.

[0032] FCN1 108 is used to estimate the noise present in mixed speech frames. Therefore, the output of FCN1 108 is a noise estimate in the speech signal (i.e., audio frame). Gammatone features are extracted from this noise and input to FCN1 108. Thus, during training, FCN1 108 learns the mapping between the gammatone function and the noise estimate. In other words, FCN1 108 generates a mask that reflects the gammatone characteristics of the noise signal. For example, the noise estimate can be an estimate of an ideal ratio mask (IRM). The IRM represents the spectral noise gain for each subband (or frequency bin) of the signal relative to the current frame. In other words, FCN1 108 can be trained to learn the mapping from the gammatone features of the training sample frame to the IRM for each subband (collectively referred to as IRM 110). Therefore, IRM 110 includes an IRM for each subband. Each mask in IRM 110 can be a 64-dimensional mask. The subband gain in a frequency bin can be the ratio of a clean audio frame (i.e., a reference audio frame) to a noisy audio frame (i.e., the current frame), or it can be an estimate of the ratio between the two. Applying IRM to noisy speech can significantly improve intelligibility. In one example, the IRM to be estimated can be given by equation (2). However, other IRM masks can also be used.

[0033]

[0034] In formula (2), S(t,f) 2 and N(t,f) 2 The parameters β and β can be used to scale the mask. For example, β can be set to 0.5.

[0035] FCN2 112 is used to estimate the speech portion (i.e., noise-free speech) present in a frame of mixed speech. Noise-free speech may include reverberation and is therefore referred to as reverberant speech. Therefore, the output of FCN2 112 is an estimate of the reverberant speech in the speech signal (i.e., the current frame). Gammatone features are extracted from this speech signal and input into FCN2 112. In other words, FCN2 112 generates a mask representing the gammatone features of the speech signal. In one example, the speech estimate can be an estimate of a phase-sensitive mask (PSM). A PSM is a time-frequency mask that implicitly contains phase-related information. A PSM is defined based on the short-time Fourier transform (STFT) amplitude of clean and noisy speech. In other words, FCN2 112 can be trained to learn the mapping from the gammatone features of a training sample frame to a PSM for each subband (collectively, PSM 114). Therefore, PSM 114 includes a PSM for each subband. Each mask in PSM 114 can be a 64-dimensional mask. The subband gain in a frequency bin can be the ratio of the clean audio frame (i.e., the reference audio frame) to the noisy audio frame (i.e., the current frame), or it can be an estimate of the ratio between the two. In one example, the PSM to be estimated can be given by formula (3). However, other PSM masks can also be used.

[0036]

[0037] In formula (3), |S(t,f)| and |Y(t,f)| represent the spectral amplitudes of the clean speech (i.e., the reference frame corresponding to the current frame) and the noisy speech (i.e., the current frame), respectively; θ represents the phase difference between the clean speech (i.e., the reference frame corresponding to the current frame) and the noisy speech (i.e., the current frame) in time-frequency units.

[0038] To reiterate, FCN1 108 is used to generate a mask for the gamma-tone features of the noise signal, and FCN2 112 is used to generate a mask for the gamma-tone features of the speech signal. FCN1 108 and FCN2 112 can be trained together using respective loss functions of mean squared error (MSE). A first loss function used to train FCN2 112 can be the MSE between the pure speech gamma-tone features of the predicted sample and the pure speech gamma-tone features of the reference sample. A second loss function used to train FCN1 108 can be the MSE between the pure noise gamma-tone features of the predicted sample and the pure noise gamma-tone features of the reference sample.

[0039] As described above, the speech gains of the ERB-divided subbands are obtained in the noise reduction stage 102. Instead of directly applying the gains to the corresponding subbands, the gains can be mapped back to the Fourier domain, such as the STFT domain, by the adaptive mapping stage 104. During the training phase, the adaptive mapping stage 104 obtains (e.g., adjusts, optimizes, etc.) the mapping parameters {β i-1,j β i,j β i+1,j}, which will be further elaborated below. The mapping parameters can be fixed parameters and can be adjusted based on experience (e.g., by trial and error), for example, by searching the possible range defined in equation (4) to minimize the convergence loss. The mapping parameters {β i-1,j β i,j β i+1,j The following conditions can be met:

[0040]

[0041] This paper describes three mapping parameters, but in practice more or fewer mapping parameters may be used. The parameter indexes (i.e., i and j) of formula (4) are further described in formula (5). Figure 5 As shown, the output of FCN2 112 (and FCN1 108) has the same dimensionality as the input to FCN2 112 (and FCN1 108). When the subband mask (i.e., PSM 114) is obtained from FCN2 112, 64 subbands are obtained because 64 gammatone features are obtained from module 116 and used as input to FCN2 112. The output of FCN2 112 is a 64-dimensional mask, i.e., PSM 114. PSM 114 is a mask corresponding to 64 subbands. However, it would be preferable to obtain the gain for each frequency bin of the spectrum (e.g., FFT spectrum), as the final output will be calculated based on this spectrum (e.g., FFT spectrum). Module 118 can be used to obtain the FFT spectrum. That is, module 118 can receive the current frame and extract the FFT spectrum from the current frame, which is the spectrum of speech with noise and reverberation. In other words, module 118 can extract the noise frequency and reverberation frequency from the current frame.

[0042] As shown in formula (5), the gain associated with each frequency bin in the subband can be estimated from the corresponding subband. Assuming that the i-th ERB subband consists of K frequency bins, the gain of the j-th frequency bin in the subband (denoted as g ij ) can be obtained by the following formula:

[0043] g ij =β i-1，j G i-1 +β i，j G i +β i+1，j G i+1 (5)

[0044] The adaptive mapping stage 104 can use the output of FCN1 108 during the training process as shown below. FCN1 108 and FCN2 112 provide amplitude gain and phase factors that will change the amplitude and phase of the gammatone feature respectively. i ) and phase factor (such as PF i ) can be combined with formula G i =MG i *PF i Get the gain of the gamma-tone feature (G i ). The adaptive mapping stage 104 may then use the gain of the gammatone feature as described in equation (5).

[0045] Figure 2 is a schematic diagram of mapping subband gains to frequency bins according to an embodiment of the present invention. Example 200 illustrates the operation process of formula (5). Example 200 illustrates a mapping pattern of gains calculated by gammatone features to a complete FFT spectrum. Example 200 includes ERB subband gains 202 of the PSM mask obtained from FCN2 112 (i.e., ERB gains or masks of PSM114) and frequency bin amplitudes of the FFT spectrum 204 obtained from module 118. Formula (5) can be used to map the mapping parameters as ERB subband gains 210 (i.e., G i ), ERB subband gain 208 (ie G i-1 ) and ERB subband gain 212 (ie G i+1 ) to calculate the gain (g) of the frequency bin 206 of the ERB subband gain 210 ij ).

[0046] Looking back Figure 1The output of the adaptive mapping stage 104 is an estimated full-band noise gain in the FFT domain, which can be used to obtain an enhanced frame. The full-band gain can be multiplied by the noisy speech (i.e., the current frame) to obtain the enhanced speech (i.e., the enhanced current frame). More specifically, the amplitude spectrum of the current frame with noise and reverberation can be multiplied by the corresponding full-band gain to obtain an enhanced frame. The enhanced frame refers to the current frame that has been denoised but still has reverberation. In this way, after the adaptive mapping stage, the estimated full-band gain can be applied to the amplitude spectrum of the noisy and reverberant speech (i.e., the current frame) to obtain an estimate of the amplitude spectrum of the reverberant speech.

[0047] The dereverberation stage 106 includes a logarithmic compression module 120 and a time-distributed deep neural network (ie, TDDNN 122 ). In some implementations, the dereverberation stage 106 may also not include the logarithmic compression module 120 .

[0048] The logarithmic compression module 120 can be used to calculate the logarithmic compression of the amplitude spectrum of the enhanced speech. The logarithmic compression module 120 can be used to reduce the dynamic range of the values ​​in the amplitude spectrum. Therefore, using logarithmic calculations instead of linear calculations can reduce the dynamic range.

[0049] During the training phase of technique 100, TDDNN 122 may be trained to map the spectrum of a predicted reverberant frame (i.e., an enhanced frame) to a clean, reverberant-free speech frame (i.e., a reference frame). For example, TDDNN 122 may receive as input a logarithmically compressed feature of a current frame. More specifically, TDDNN 122 receives a logarithmically compressed estimate of the magnitude spectrum of an enhanced frame (e.g., a frame that has been denoised but still has reverberation). The output of TDDNN 122 is an estimate of the spectrum of a clean frame. A clean frame is a frame after the current frame has been denoised and dereverberated.

[0050] In one example, the TDDNN 122 can be a time-distributed fully connected deep neural network with three hidden layers and a linear output layer. It is well known that a time-distributed neural network maintains an internal state representing previously input context information for a period of time that is not determined a priori but depends on the weights and input data obtained during the training process. It is also well known in the industry that a time-distributed network can convert an input sequence into an output sequence while taking into account the context information (i.e., the internal state). Therefore, there is no need to explicitly input the context information (such as features of one or more frames before the current frame) to the TDDNN 122 along with the current frame. Therefore, the complexity of the TDDNN 122 can be reduced by reducing the input to the TDDNN 122.

[0051] Figure 3300 is an example diagram of a technique for noise reduction and dereverberation of real-time audio, drawn according to an embodiment of the present invention. Technique 300 can be used in real-time applications such as audio and / or video conferencing, telephone calls, etc. to reduce noise and dereverberation of speech exchanged between communicating parties or speech transmitted from a sending user to one or more receiving users. Technique 300 can be implemented by a sending device. Thus, before being sent to the receiving user, the speech from the speaker can be subjected to noise reduction and dereverberation on the sending user's device, so that the receiving user can receive clean speech (i.e., speech after noise reduction and dereverberation). Technique 300 can also be implemented by a receiving device. Therefore, the speech received at the receiving user's receiving device may be noisy and reverberant. Technique 300 can be run on the received speech to produce clean speech (i.e., speech after noise reduction and dereverberation). The clean speech is then output to the receiving user. Technique 300 can also be implemented by a central server that receives mixed speech from a sending device, executes technique 300 to obtain clean and reverberation-free speech, and sends (e.g., forwards, transmits, etc.) the clean and reverberation-free speech to one or more recipients. Technique 300 can be implemented on a computing device, such as Figure 4 computing devices in .

[0052] The stages included in Technology 300 are Figure 1 The stages of the technique 100 shown in FIG are similar. Thus, the technique 300 includes a feature extraction stage comprising modules 316 and 318, which are similar to the Figure 1 Modules 116 and 118 are similar in FIG. 3 ; a noise reduction stage 302 ; an adaptive mapping stage 304 ; and a dereverberation stage 306 .

[0053] Technology 300 Use Figure 1 Some components of the technique 100 are trained, namely the denoising stage 302 uses FCN2312, which uses Figure 1 The parameters of the trained FCN2 112 are shown; the adaptive mapping stage 304 uses the parameters of Figure 1 The mapping parameters adjusted by the adaptive mapping 104; the logarithmic compression module 320 used in the dereverberation stage 306 is as follows Figure 1 As shown in the logarithmic compression module 120; and TDDNN 322, which also uses Figure 1 Parameters of the trained TDDNN 122 are shown.

[0054] Without loss of generality, an embodiment is used as an example: the technology 300 can be described as performing the following steps: 1) extracting 64-channel gammatone features for each current frame of the received mixed speech (i.e., speech with noise and reverberation); 2) using the trained FCN2 to perform speech estimation; 3) using adaptive gain mapping to obtain the predicted reverberant speech spectrum in the current frame; 4) performing logarithmic compression on the predicted reverberant speech spectrum; 5) using a time-distributed fully connected deep neural network (i.e., TDDNN 322) to estimate the clean speech spectrum in each current frame.

[0055] Technique 300 utilizes Figure 1 The improved neural network and frozen parameters of technique 100 can effectively achieve both single-channel dereverberation and noise reduction. Technique 300 can be implemented frame by frame and in a causal manner. A 64-channel gamma-tone feature can be extracted for the current frame and concatenated with the corresponding feature in the context window to form the input of FCN2 312. For the gamma-tone filter, as defined in equation (1), each gamma-tone feature can be obtained by multiplying the FFT spectrum with the context window.

[0056] The frozen parameters {β i-1,j β i,j β i+1,j The predicted ERB subband gains are mapped to the STFT domain. The product of the current frame's mixed speech spectrum and the predicted gains is used as an estimate of the reverberant speech spectrum. After logarithmic compression, the estimated speech spectrum for the current frame is then used as input to a time-distributed, fully connected deep neural network to implement dereverberation.

[0057] It should be noted that Figure 1 The adaptive mapping stage 104 of the 300 (i.e., during training) receives input data from both FCN1 108 and FCN2 112. However, during inference (i.e., in real-time applications), the technique 302 does not use the components corresponding to FCN1 108. In the technique 300, only FCN2 312 is used. FCN 312 is used for dereverberation. The output of FCN2 312 is directly used as the input to the adaptive mapping stage 312 (G i =PF i ).

[0058] Figure 4 4 is a schematic block diagram of a computing device according to an embodiment of the present invention. Computing device 400 may be a computing system including multiple computing devices, or a single computing device such as a mobile phone, tablet computer, laptop computer, notebook computer, desktop computer, etc.

[0059] The processor 402 in the computing device 400 can be a conventional central processing unit. The processor 402 can also be another type of device or multiple devices capable of manipulating or processing existing or later developed information. For example, although the examples herein may be implemented with a single processor (such as the processor 402), using multiple processors may provide advantages in speed and efficiency.

[0060] In one implementation, the memory 404 in the computing device 400 may be a read-only memory (ROM) device or a random access memory (RAM) device. Other appropriate types of storage devices may also be used as the memory 404. The memory 204 may contain code and data 406 accessed by the processor 402 using a bus 412. The memory 404 may also contain an operating system 408 and applications 410, wherein the applications 410 include at least one program that allows the processor 402 to perform one or more of the techniques described herein. For example, the applications 410 may include applications 1 through N, which include programs and techniques useful in implementing simultaneous dereverberation and noise reduction applications. For example, the applications 410 may include technique 100 or its various techniques to implement a training phase. For example, the applications 410 may include technique 300 or its various techniques to implement real-time dereverberation and noise reduction functionality. The computing device 400 may also include a secondary storage device 414, such as a memory card used with mobile computing devices.

[0061] Computing device 400 may also include one or more output devices, such as display 418. In one example, display 418 may be a touch-sensitive display that combines a display with touch-sensitive elements operable for touch input. Display 418 may be coupled to processor 402 via bus 412. Other output devices that allow a user to program or use computing device 400 may also be used in addition to or in place of display 418. If the output device is or includes a display, the display may be implemented in various ways, including a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0062] The computing device 400 may also include an image sensing device 420 (e.g., a camera), or any other image sensing device 420 now known or later developed that can sense an image (e.g., an image of a user operating the computing device 400), or communicate with the image sensing device 420. The image sensing device 420 may be positioned to face the user operating the computing device 400. In one example, the position and optical axis of the image sensing device 420 may be configured such that the field of view includes an area directly adjacent to and visible to the display 418.

[0063] The computing device 400 may also include a sound sensing device 422 (e.g., a microphone), or any other sound sensing device 422 now available or later developed that can sense sounds near the device 400, or communicate with the sound sensing device 422. The sound sensing device 422 may be positioned facing a user operating the computing device 400 and may be configured to receive sounds, such as sounds emitted by the user when the user operates the computing device 400, such as speech or other sounds. The computing device 400 may also include or communicate with a sound playback device 424, such as a speaker, headphones, or any other sound playback device now available or later developed that can play sounds in accordance with instructions from the computing device 400.

[0064] Figure 4 The processor 402 and memory 404 of the computing device 400 are depicted as being integrated into a single processing unit, although other configurations are possible. The operations of the processor 402 may be distributed across multiple machines (each machine containing one or more processors), which may be directly coupled or coupled across a local or other network. The memory 404 may be distributed across multiple machines, such as network-based storage or storage in multiple machines that run the operations of the computing device 400. While only a single bus is described herein, the bus 412 of the computing device 400 may also be comprised of multiple buses. Furthermore, the auxiliary memory 414 may be directly coupled to other components of the computing device 400, may be accessed via a network, or may comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, the computing device 400 may be implemented in a variety of configurations.

[0065] Figure 5 This is a structural example diagram of a deep feature convolutional neural network (CNN) drawn according to an embodiment of the present invention. Figure 1 FCN1 108 and FCN2 112, and Figure 3 The FCN2 312 in the embodiment may have the structure shown in example 500. However, other structures are also possible.

[0066] In a CNN, the feature extraction phase typically involves a set of convolution operations, typically consisting of a series of filters (assuming a filter size of k squared) that filter the input data (e.g., an image, an audio frame of audio samples, etc.). For example, in machine vision applications, these filters can be used to find certain features in the input image. These features can include edges, corners, endpoints, and so on. As multiple layers of convolution operations are stacked, subsequent convolution operations can find higher-level features.

[0067] In a CNN, the classification part is typically a set of fully connected layers. You can think of a fully connected layer as examining all the input features of an audio frame to generate a high-level classifier. This high-level classifier, passed through several stages (like a series), ultimately produces the desired classification output.

[0068] As mentioned above, a typical CNN network consists of many convolution operations (such as the feature extraction part) followed by many fully connected layers. The number of operations of each type and their respective sizes are usually determined during the training phase of machine learning. As is well known to those skilled in the art, additional layers and / or operations can also be included in each part. For example, a combination of pooling, max pooling, dropout, activation, normalization, batch normalization and other operations can be used in conjunction with convolution operations (i.e., operations in the feature extraction part) and / or fully connected operations (i.e., operations in the classification part). The fully connected layer can be called a densely connected layer. As is well known to those skilled in the art, the convolution operation can use SeparableConvolution2D or Convolution2D operations.

[0069] A convolutional layer can be a set of operations that starts with a Convolution2D or SeparableConvolution2D operation, followed by zero or more operations (e.g., Pooling, Dropout, Activation, Normalization, BatchNormalization, other operations, or a combination of the above) until it reaches another convolutional layer, a dense layer, or the output of a CNN. Similarly, a dense layer can be a set of operations or layers that starts with a dense operation (i.e., a fully connected layer), followed by zero or more operations (e.g., Pooling, Dropout, Activation, Normalization, BatchNormalization, other operations, or a combination of the above) until it reaches another convolutional layer, a dense layer, or the output of a CNN. The distinction between feature extraction based on convolutional networks and feature classification using dense operations can be demarcated by the flattening operation, which flattens the multidimensional matrix used in feature extraction into a vector.

[0070] In a typical CNN, each convolutional layer consists of a set of filters. While a filter is applied to a subset of the input data at a time, it can be applied to the entire input through operations such as sweeping the input signal. The operations performed by this layer are typically linear or matrix multiplications. The activation function can be linear or nonlinear (e.g., sigmoid, arcTan, tanH, ReLu, etc.).

[0071] Each fully connected operation is a linear operation where each input is connected to each output through a weight (also called a parameter). Thus, a fully connected layer with N inputs and M outputs can have a total of N×M weights. As mentioned above, a dense operation is usually followed by a nonlinear activation function to generate the output of the layer.

[0072] Example 500 includes an input layer 502. Example 500 also includes three strided convolutional layers (i.e., layers 504, 506, 508) for downsampling and three transposed convolutional layers (i.e., layers 510, 512, 514) for upsampling. As the receptive field increases, global information is extracted in both the time domain and the frequency domain. The receptive field can be defined as the area in the input space that a particular feature is observing and / or being affected by.

[0073] Example 500 is an encoder-decoder architecture. Here, the downsampling path compresses the input gamma-tone spectrum into a low-resolution feature map, thereby modeling longer temporal and frequency dependencies. The upsampling path takes this compact representation as input and increases the resolution to map the output back to the original size.

[0074] To mitigate information loss caused by compression in the encoder and allow for direct propagation of gradients, connections 516 and 518 can be skipped and layers of the same size can be directly connected. Leaky rectified linear units (ReLU) and sigmoid functions can be used as activation functions for the hidden and output layers, respectively. However, other activation functions can also be used.

[0075] Figure 6 FIG6 is a flowchart illustrating a technique for performing noise reduction and dereverberation on an audio sample according to an embodiment of the present invention. In one example, the technique 600 may receive an audio sample, such as an audio stream, which may be part of a video stream. In another example, the technique 600 may receive frames of an audio stream for processing. In yet another example, the technique 600 may divide the audio sample into frames and process the audio sample according to the frames. Figure 3 The technique 300 in processes each frame separately, as described below.

[0076] The technique 600 can be performed by a computing device such as Figure 4Technique 600 may be implemented as a software program executed by a computing device (such as computing device 400). The software program may include machine-readable instructions that may be stored in a memory (such as memory 404 or secondary memory 414) and that, when executed by a processor (such as processor 402), cause the computing device to perform technique 600. Technique 600 may be implemented using dedicated hardware or firmware. Multiple processors and / or multiple memories may also be used.

[0077] At 602, technique 600 extracts spectral features representing a cochlear impulse response of the auditory system from a frame of audio samples. In one example, the spectral features can be extracted using a gammatone filter bank, as described above.

[0078] At 604, technique 600 obtains an estimate of the speech signal in the frame from a neural network. The neural network may be as follows: Figure 3 As shown in FCN2 314. Therefore, in Figure 1 The training phase shown trains the neural network to accept spectral features as input and output an estimate of the speech signal. As described above, the estimate of the speech signal can include reverberation, but not noise present in the frame. In one example, the estimate of the speech signal output from the neural network is an estimate of a phase-sensitive mask (PSM), as described above. Thus, the estimate of the phase-sensitive mask can include predicted equivalent rectangular bandwidth (ERB) subband gains.

[0079] In one example, obtaining an estimate of the speech signal in the frame from the neural network may include concatenating spectral features with corresponding features of the context window to obtain an input vector; and providing the input vector as input to the neural network.

[0080] At 606, technique 600 maps the estimate of the speech signal to the frequency domain of the frame using the mapping parameters obtained in the training phase to obtain an estimate of the reverberant speech spectrum in the frame, such as Figure 3 The adaptive mapping stage 304 in FIG. 3 is shown. The frequency domain refers to the Fourier domain. In one example, the frequency domain may include short-time Fourier transform (STFT) features extracted from the frame.

[0081] Mapping the estimate of the speech signal to the frequency domain of the frame using the mapping parameters may include estimating the gain g for each frequency bin within the subband using equation (5) ij .

[0082] At 608, the technique 600 obtains a dereverberated frame for the frame from a time-distributed neural network. The estimate of the reverberant speech spectrum in the frame is used as input to the time-distributed neural network. The time-distributed neural network may be as follows: Figure 3In one example, the technique 600 uses logarithmic compression to change the dynamic range, as shown in FIG. Figure 3 Therefore, obtaining the dereverberation frame of the frame from the time-distributed neural network may include: performing logarithmic compression on the reverberation speech spectrum in the frame to obtain a reverberation speech spectrum after logarithmic compression operation; and using the reverberation speech spectrum after logarithmic compression operation as input to the time-distributed neural network.

[0083] The present invention also includes a system for performing noise reduction and dereverberation on audio samples, which includes a noise reduction module, an adaptive mapping module, and a dereverberation module.

[0084] like Figure 3 As shown, the noise reduction module can be configured to extract spectral features from a frame of audio samples; and obtain an estimate of the speech signal in the frame from a trained first neural network during a training phase, wherein the first neural network is trained to accept spectral features as input and output an estimate of the speech signal. As also described above, the adaptive mapping module can be configured to map the estimate of the speech signal to the frequency domain of the frame using the mapping parameters obtained during the training phase to obtain an estimate of the reverberant speech spectrum in the frame. As described above, the dereverberation module can be configured to obtain a dereverberated frame of the frame from a time-distributed neural network, wherein the estimate of the reverberant speech spectrum in the frame serves as input to the time-distributed neural network.

[0085] During the training phase, Figure 1 As shown in the adaptive mapping stage 104 in FIG, the adaptive mapping module can be trained step by step to receive the amplitude spectrum of the training frame of the training audio sample, wherein the amplitude spectrum includes noise and reverberation frequencies; receive the estimated value of the training speech signal from the noise reduction module; receive the estimated value of the training noise signal from the noise reduction module; and adjust the mapping parameters for gain mapping according to the amplitude spectrum. Figure 1 As shown in TDDNN 122 in , the dereverberation module can be gradually trained to logarithmically compress the amplitude spectrum of the reverberant speech spectrum; and the time distributed neural network can be trained to map the logarithmically compressed amplitude spectrum of the reverberant speech spectrum to a pure undisturbed speech spectrum.

[0086] Embodiments of the present invention also encompass techniques for denoising and dereverberation of an audio (e.g., speech) stream. The techniques include receiving an audio stream; performing the following operations on at least a portion of frames in the audio stream: extracting gamma-tone features of a current frame of the frames in the audio stream; inputting the gamma-tone features into a neural network that estimates ERB subband gains in the current frame; obtaining a magnitude spectrum of the current frame; mapping the estimated ERB subband gains into the magnitude spectrum to obtain an estimate of a reverberant speech spectrum in the current frame; applying a nonlinear function to the reverberant speech spectrum in the current frame to obtain a compressed reverberant speech spectrum; and inputting the compressed reverberant speech spectrum into a time-distributed neural network to obtain a dereverberated current frame.

[0087] To simplify the explanation, Figure 1 , 3 and 6, respectively, are illustrated as a series of modules, steps, or operations. However, according to the present invention, these modules, steps, or operations may occur in various orders and / or simultaneously. In addition, other steps or operations not mentioned and described herein may also be used. Furthermore, the techniques designed in accordance with the present invention may not require all of the steps or operations shown to be implemented.

[0088] The word "example" is used herein to mean an example, instance, or illustration. Any feature or design described herein as an "example" is not necessarily superior or preferable to other features or designs. Instead, the word "example" is used to present concepts in a concrete way. The word "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or." That is, "X includes A or B" is intended to mean any natural inclusive permutation, unless otherwise specified or clear from the context. In other words, if X includes A, X includes B, or X includes A and B, then "X includes A or B" holds true in any of the aforementioned instances. Furthermore, throughout this application and the appended claims, "a" or "an" should generally be interpreted to mean "one or more," unless otherwise specified or the context clearly indicates the singular form. Furthermore, throughout this document, the phrases "a feature" or "a function" do not imply the same embodiment or function, unless specifically stated otherwise.

[0089] Figure 4 The computing device 400 shown and / or any components thereof and Figure 1 or Figure 3Any modules or components shown (as well as the techniques, algorithms, methods, instructions, etc. stored thereon and / or executed thereby) may be implemented using hardware, software, or any combination thereof. Hardware includes, for example, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, firmware, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuitry. Throughout the present invention, the term "processor" should be understood to encompass any combination of one or more of the foregoing. Terms such as "signal" and "data" may be used interchangeably.

[0090] Furthermore, the techniques may be implemented using a general-purpose computer or processor with a computer program that, when executed, executes any corresponding technique, algorithm, and / or instruction described herein. Alternatively, a dedicated computer or processor equipped with dedicated hardware may be used to implement any method, algorithm, or instruction described herein.

[0091] Furthermore, all or part of the embodiments of the present invention may take the form of a computer program product that can be used by a computer or accessed by a computer-readable medium. A computer-usable or computer-readable medium can be any device that can contain, store, communicate, or transport a program or data structure for use by or in connection with any processor. The medium can be an electronic, magnetic, optical, electromagnetic, or semiconductor device, among others. Other suitable media may also be included.

[0092] Although the present invention has been described in conjunction with certain embodiments, it should be understood that the invention is not limited to the disclosed embodiments. On the other hand, the present invention is intended to cover various modifications and equivalent arrangements within the scope of the claims, which should be given the broadest interpretation to cover all such modifications and equivalent arrangements permitted by law.

Claims

1. A method for denoising and dereverberation of audio samples: extracting spectral features of a cochlear impulse response of the auditory system from frames of audio samples; obtaining an estimate of the speech signal in the frame from a neural network that has been trained during a training phase to accept spectral features as input and output an estimate of the speech signal, wherein the estimate of the speech signal includes reverberation but excludes noise present in the frame; Using the mapping parameters obtained in the training phase, performing a weighted summation on an estimate of the speech signal output from the neural network, wherein the estimate of the speech signal is an estimate of a phase-sensitive mask, and the estimate of the phase-sensitive mask includes predicted equivalent rectangular bandwidth (ERB) subband gains; mapping the ERB subband gains to corresponding frequency bins through weighted summation to calculate a gain for each frequency bin in the frame; and multiplying the gains of the frequency bins by the mixed speech spectrum of the frame to obtain an estimate of the reverberant speech spectrum in the frame; as well as A dereverberated frame of the frame is obtained from a time-distributed neural network, wherein an estimate of the reverberant speech spectrum in the frame is used as an input to the time-distributed neural network. The method according to claim 1 , wherein the spectral features are extracted by a gammatone filter bank. The method of claim 1 , wherein the frequency domain comprises short-time Fourier transform (STFT) features extracted from the frame.

4. The method of claim 1 , wherein obtaining an estimate of the speech signal in the frame from a neural network comprises: Concatenate the spectral features with the corresponding features of the context window to obtain the input vector; as well as The input vector is given as input to the neural network.

5. The method of claim 1 , wherein obtaining a dereverberated frame of the frame from a time-distributed neural network comprises: Performing logarithmic compression on the reverberation speech spectrum in the frame to obtain a logarithmically compressed reverberation speech spectrum; as well as The logarithmically compressed reverberant speech spectrum is used as the input of the time-distributed neural network.

6. The method according to claim 1, wherein a weighted summation is performed on an estimate of a speech signal output from a neural network using mapping parameters obtained during a training phase, wherein the estimate of the speech signal is an estimate of a phase-sensitive mask, the estimate of the phase-sensitive mask including predicted equivalent rectangular bandwidth (ERB) subband gains, mapping the ERB subband gains to corresponding frequency bins via weighted summation to calculate a gain for each frequency bin in the frame, and multiplying the gains of the frequency bins by the mixed speech spectrum of the frame to obtain an estimate of the reverberant speech spectrum in the frame, wherein the method comprises: The gain g of each frequency bin in the subband is estimated by the following formula: ij g ij =b i-1，j G i-1 +b i，j G i +b i+1，j G i+1 , where g ij represents the gain of the jth frequency bin in the i-th ERB subband, Among them G i-1 , G i and G i+1 denote the gains of the (i-1)th, i-th and (i+1)th sub-bands respectively, where {β i-1，j β i,j β i+1，j } is the mapping parameter.

7. A system for performing noise reduction and dereverberation on audio samples, comprising: Noise reduction module; Adaptive mapping module; and De-reverberation module, Configure the Noise Reduction module to do the following: extracting spectral features of a cochlear impulse response of the auditory system from the frames of audio samples; and obtaining an estimate of the speech signal in the frame from a neural network that was trained during a training phase to accept spectral features as input and output an estimate of the speech signal that includes reverberation but excludes noise present in the frame; Configure the adaptive mapping module to do the following: Using the mapping parameters obtained in the training phase, performing a weighted summation on an estimate of the speech signal output from the neural network, wherein the estimate of the speech signal is an estimate of a phase-sensitive mask, and the estimate of the phase-sensitive mask includes predicted equivalent rectangular bandwidth (ERB) subband gains; mapping the ERB subband gains to corresponding frequency bins through weighted summation to calculate a gain for each frequency bin in the frame; and multiplying the gains of the frequency bins by the mixed speech spectrum of the frame to obtain an estimate of the reverberant speech spectrum in the frame; as well as Configure the De-Reverberation module to do the following: A dereverberated frame of the frame is obtained from a time-distributed neural network, wherein an estimate of the reverberant speech spectrum in the frame is used as an input to the time-distributed neural network.

8. The system of claim 7, wherein the denoising module is gradually trained in the training phase to perform the following operations: extracting training spectral features from training frames of training audio samples; Using the training spectral features, a first neural network is trained to estimate a training speech signal contained in the training audio samples; and A second neural network is trained using the training spectral features to estimate the training speech signal contained in the training audio samples.

9. The system according to claim 8, wherein The training spectrum features include gamma-tone features extracted using a 64-way filter bank. where the training speech signal is estimated by a phase-sensitive mask, and The training noise signal is estimated by the ideal soft threshold mask.

10. The system of claim 8, wherein the adaptive mapping module is gradually trained in a training phase to perform the following operations: receiving a magnitude spectrum of a training frame of a training audio sample, wherein the magnitude spectrum includes noise frequencies and reverberation frequencies; receiving an estimate of a training speech signal from a noise reduction module; Receive an estimate of the training noise signal from the denoising module; as well as Mapping parameters for gain mapping are adjusted according to the magnitude spectrum.

11. The system according to claim 10, wherein the dereverberation module is gradually trained in the training phase to perform the following operations: logarithmically compressing the amplitude spectrum of the reverberant speech spectrum to obtain a logarithmically compressed reverberant speech spectrum; and The time-distributed neural network is trained to map the logarithmically compressed reverberant speech spectrum to the pure undisturbed spectrum. The loudest speech spectrum.

12. A set of equipment for performing noise reduction and dereverberation on audio samples, comprising: A processor configured to: extracting spectral features of a cochlear impulse response of the auditory system from frames of audio samples; obtaining an estimate of the speech signal in the frame from a neural network that was trained during a training phase to accept spectral features as input and output an estimate of the speech signal that includes reverberation but excludes noise present in the frame; Using the mapping parameters obtained in the training phase, performing a weighted summation on an estimate of the speech signal output from the neural network, wherein the estimate of the speech signal is an estimate of a phase-sensitive mask, and the estimate of the phase-sensitive mask includes predicted equivalent rectangular bandwidth (ERB) subband gains; mapping the ERB subband gains to corresponding frequency bins through weighted summation to calculate a gain for each frequency bin in the frame; and multiplying the gains of the frequency bins by the mixed speech spectrum of the frame to obtain an estimate of the reverberant speech spectrum in the frame; as well as A dereverberated frame of the frame is obtained from a time-distributed neural network, wherein an estimate of the reverberant speech spectrum in the frame is used as an input to the time-distributed neural network. The apparatus according to claim 12 , wherein the spectral features are extracted by a gammatone filter bank.

14. The apparatus of claim 12, wherein the frequency domain comprises Short Time Fourier Transform (STFT) features extracted from the frame.

15. The apparatus of claim 12 , wherein obtaining an estimate of the speech signal in the frame from a neural network comprises: Concatenate the spectral features with the corresponding features of the context window to obtain the input vector; as well as The input vector is given as input to the neural network.

16. The apparatus of claim 12, wherein obtaining the dereverberated frame of the frame from a time-distributed neural network comprises: Performing logarithmic compression on the reverberation speech spectrum in the frame to obtain a logarithmically compressed reverberation speech spectrum; as well as The logarithmically compressed reverberant speech spectrum is used as the input of the time-distributed neural network.

Images