Method, program, system, and non-transitory computer-readable medium
A deep neural network architecture that integrates silence interval detection and noise removal components addresses the challenge of non-stationary noise in speech recordings, achieving enhanced denoising performance and speech quality.
Patent Information
- Application Number
- JP2023530195
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-11-20
- Filing Date
- 2021-07-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-07-20
AI Technical Summary
Existing speech denoising techniques struggle to effectively remove non-stationary noise from voice recordings, particularly due to the difficulty in identifying and utilizing silence intervals, which are crucial for capturing time-varying noise profiles.
A deep neural network architecture that integrates silence interval detection, noise estimation, and noise removal components, utilizing LSTM structures to detect and process silence intervals in speech recordings, enabling robust denoising of non-stationary noise.
The proposed method achieves superior denoising performance by automatically identifying silence intervals, effectively reducing noise even in challenging conditions, outperforming state-of-the-art methods and maintaining speech quality across various signal-to-noise ratios.
Smart Images

Figure 0007588720000050 
Figure 0007588720000051 
Figure 0007588720000052
Abstract
Description
[Technical field]
[0001] This invention was made with Government support under Grant Nos. 1910839, 1453101, and 1850069 awarded by the National Science Foundation and contracts awarded by the Knowledge-directed Artificial Intelligence Reasoning Over Schemas (KAIROS) program administered by the Defense Advanced Research Projects Agency (DARPA). The Government has certain rights in this invention. [Background technology]
[0002] Recordings of human speech are often contaminated with noise from a variety of sources. Some noise in a recording may be stationary, while other noise may vary in frequency and amplitude during the recording. This latter noise, called non-stationary noise, is difficult to remove from a recording. [Brief description of the drawings]
[0003] The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Like reference numbers designate corresponding parts throughout the different views. In the figures of the accompanying drawings, embodiments are illustrated by way of example, and not by way of limitation. [Figure 1] Network configuration. [Diagram 2] Silent intervals over time. [Diagram 3] Examples of intermediate and final results [Figure 4] Noise Gallery [Diagram 5] Quantitative comparison [Figure 6] The quality of the noise reduction with respect to the input SNR. [Figure 7] Noisy audio constructed based on different SNR levels [Figure 8] Noise reduction quality at different input SNRs [Figure 9] Silence Interval Detection Example DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0004] Systems, methods, and other implementations (including hardware, software, and hybrid hardware / software implementations) are disclosed that are directed in the framework of speech denoising that exploit the abundance of silent intervals in speech to train models for automatic speech denoising given only mono channel audio. The implementations described herein are based on deep neural networks for speech denoising techniques that tightly integrate silent intervals, thereby overcoming many of the limitations of classical techniques. The goal is not to only identify a single silent interval, but to find as many possible silent intervals over time as possible. In fact, silent intervals in speech appear to be abundant: psycholinguistic studies show that there is almost always a pause after each sentence, and even after each word in an utterance. Each pause, no matter how short, provides a silent interval-specific noise feature that is local in time. Collectively, these silent intervals incorporate a time-varying picture of the background noise, allowing the neural network to better denoise speech signals, even in the presence of non-stationary noise.
[0005] The techniques described herein utilize a neural network configuration based on a long short-term memory (LSTM) structure to reliably denoise voice recordings (other learning configurations / structures can also be used). To do so, the LSTM is trained on noise acquired from intermittent gaps in speech, called silent intervals, which it automatically identifies in the recordings. Silent intervals contain a combination of stationary and non-stationary noise, so the spectral distribution of noise during these silent intervals can be utilized in denoising. The LSTM can remove stationary and non-stationary spectrum in the voice intervals, resulting in robustly denoised high quality voice recordings. The technique is also applicable in audio recording, film production, and speech-to-text applications.
[0006] To interleave a neural network with an established noise removal pipeline, a network architecture is proposed that includes three main components (shown in Figure 1): i) a component dedicated to silence interval detection, ii) another component that estimates the full noise from that manifested in the silence intervals, similar to the inpainting process in computer vision, and iii) another component that cleans up the input signal.
[0007] More specifically, the silence interval detection component is configured to detect silence intervals in an input signal. The input to this component is the spectrogram of the input (noisy) signal x. The spectrogram S x is first encoded into a 2D feature map by a 2D convolutional encoder, which is then processed by a bidirectional LSTM followed by two fully connected (FC) layers. The bidirectional LSTM is suitable for processing the time series features resulting from the spectrogram, and the FC layers apply the features of each time sample to adapt to the variable length input. The output from this network component is a vector D(S x ). D(S xEach element of D(S) is a scalar in [0,1] (after application of a Sigmoid function) that indicates the confidence score of a small time bin that is silent. In some examples, each time bin has a duration of 1 / 30th of a second, which is small enough to capture short speech interruptions, yet large enough to allow robust prediction. The output vector D(S x ) is then expanded into a longer mask, denoted m(x), whose elements indicate the confidence with which to classify each sample of the input signal x as pure noise.
number
number
[0008] In the noise estimation component / module, the resulting noise from silent interval detection
number
number
number
number
number
[0009] Finally, the noise from the input signal x is cleaned up by utilizing a noise removal component / module. The neural network R takes as input the input speech spectrogram S x , and the estimated full noise spectrogram
number
number
number
number
number
[0010] Since subgradients exist at every stage, in some embodiments the network can be trained in an end-to-end manner with a stochastic gradient descent approach. The following loss function is optimized:
number
number
number
[0011] While producing plausible denoising results, the end-to-end training process is unsupervised for silent interval detection: only the loss function is responsible for restoring the noise and clean speech signals. However, somewhat surprisingly, the ability to detect silent intervals is automatically produced as the output of the first network component. In other words, the network automatically learns to detect silent intervals for speech denoising without this supervision.
[0012] As the model is learning to detect silence intervals on its own, the silence detection can be directly monitored to further improve the quality of the denoising. To that end, a term can be added to the loss function above that penalizes the discrepancy between the detected silence intervals and their ground truth. Experiments have shown that this method is not effective, but instead the model is trained in two successive stages. First, the silence interval detection component is computed through the following loss function:
number
number
[0013] Next, the noise estimation and removal component is trained with the loss function L0. This training phase starts by ignoring the silence detection component. In the loss function L0, the spectrogram of noise exposed to the estimated silence intervals is
number
number
[0014] Thus, in some embodiments, a system is provided that includes a receiver unit (e.g., a microphone, a communication module for receiving an electronic signal representation of audio / sound, etc.) that receives an audio signal representation, and a controller (e.g., a programmable device) that implements one or more learning engines and is in communication with the receiver unit and a memory device that stores programmable instructions to utilize a first learning model to detect one or more silence intervals in the received audio signal representation in which the foreground sound level is reduced, determine an estimated full noise profile corresponding to the audio signal representation based on the detected one or more silence intervals, and generate a resultant audio signal representation having a reduced noise level based on the received audio signal representation and the determined estimated full noise profile using a second learning model. In some implementations, a non-transitory computer-readable medium is provided that stores a set of instructions executable on at least one programmable device for receiving an audio signal representation, utilizing a first learning model to detect one or more silence intervals in the received audio signal representation with a reduced foreground sound level, determine an estimated full noise profile corresponding to the audio signal representation based on the detected one or more silence intervals, and generate a resultant audio signal representation having a reduced noise level based on the received audio signal representation and the determined estimated full noise profile using a second learning model.
[0015] In some implementations, a method is provided that includes receiving an audio signal representation, utilizing a first learning model to detect one or more silence intervals in the received audio signal representation with a reduced foreground sound level, determining an estimated full noise profile corresponding to the audio signal representation based on the detected one or more silence intervals, and generating a resultant audio signal representation having a reduced noise level based on the received audio signal representation and the determined estimated full noise profile using a second learning model.
[0016] In some examples, detecting one or more silence intervals utilizing the first learning model may include dividing the audio signal representation into a plurality of segments, each segment being shorter than a length of an interval in the received audio signal representation, converting the plurality of segments into a time-frequency representation, and implementing the first learning model to process the time-frequency representation of the plurality of segments utilizing the first learning machine to generate, for each of the plurality of segments, a noise vector that includes a representation of a confidence value of the likelihood that each one of the plurality of segments is a silence interval. In such examples, processing the time-frequency representation may include encoding the time-frequency representation of the plurality of segments with a 2D convolutional encoder to generate a 2D feature map, applying a learning network structure comprising at least a bidirectional long short-term memory (LSTM) structure to the 2D feature map to generate a silence vector, determining a noise mask from the silence vector, and generating a partial noise profile for the audio signal representation based on the audio signal representation and the noise mask.
[0017] In some embodiments, determining the estimated full noise profile may include generating a partial noise profile representative of time-frequency characteristics of the detected silence interval or intervals, converting the audio signal representation and the partial noise profile to respective time-frequency representations, applying convolutional encoding to the time-frequency representation of the audio signal representation and the partial noise profile to generate an encoded audio signal representation and an encoded partial noise profile, and combining the encoded audio signal representation and the encoded partial noise profile to generate the estimated full noise profile. In some examples, generating a resulting audio signal representation having a reduced noise level may include generating a time-frequency representation of the audio signal representation and the estimated full noise profile, and applying a second learning model to the audio signal representation and the time-frequency representation of the estimated full noise profile to generate a resulting audio signal representation. The second learning model may be implemented by a bidirectional long short-term memory (LSTM) structure.
[0018] As noted, implementations of the denoising processes described herein may be realized using one or more learning machines (such as neural networks). Neural networks are generally composed of multiple layers of linear transformations (multiplication by a matrix of "weights"), each followed by a nonlinear function (e.g., modified linear activation function, or ReLU, etc.). The linear transformations are learned during training by making small changes to the weight matrix that gradually make the transformation more useful for the final classification task (or other type of desired output). Layered networks may include convolutional processes, followed by pooling processes, with intermediate connections between layers that improve information sharing between layers. Some examples of learning engine techniques / configurations that can be used include generating automatic encoders and using dense layers of the network to correlate with the probability of future events via support vector machines, or building regression or classification neural network models that predict certain outputs from input data (based on training that reflects correlations between similar inputs and predicted outputs).
[0019] Examples of neural networks include convolutional neural networks (CNNs), feedforward neural networks, recurrent neural networks (RNNs, e.g., implemented using long short-term memory (LSTM) structures), and the like. A feedforward network includes one or more layers of learning nodes / elements that connect to one or more portions of input data. In a feedforward network, the connections of the inputs and layers of learning elements are such that the input data and intermediate data propagate forward toward the output of the network. Typically, there are no feedback loops or cycles in the configuration / structure of a feedforward network. Convolutional layers allow the network to efficiently learn features by applying the same learned transformations to subdivisions of data. In some embodiments, the various learning processes performed through the use of a learner can be realized using keras (an open source neural network library) building blocks and / or NumPy (an open source programming library useful for implementing modules that process arrays) building blocks.
[0020] In some embodiments, various learning engine implementations may include a trained learning engine (e.g., neural network) and a corresponding coupled learning engine controller / adapter configured to determine and / or adapt the learning engine parameters (e.g., neural network weights) that will generate the desired output. In such implementations, the training data includes a set of input records along with corresponding data that defines the ground truth for the input training records. After initial training of various learning engines including the systems described herein, subsequent training may be performed intermittently (regularly or irregularly). Upon completion of a training cycle by an adapter / controller coupled to a particular learning engine, the adapter provides data representative of updates / changes (e.g., in the form of parameter values / weights assigned to the links of a neural network-based learning engine) to the particular learning engine, causing the learning engine to update in response to the completed training cycle.
[0021] Execution of the various techniques and operations described herein may be facilitated by a controller device (e.g., a processor-based computing device) that may be implemented as part of an audio communication device (such as a hearing aid device). Such a controller device may include a processor-based device, such as a computing device that typically includes a central processing unit or processing core. The device may also include one or more dedicated learning machines (e.g., neural networks), which may be part of the CPU or processing core. In addition to the CPU, the system includes a main memory, a cache memory, and a bus interface circuit. The controller device may include a mass storage element, such as a hard drive (solid-state hard drive or other type of hard drive) or a flash drive associated with the computer system. The controller device may further include a keyboard, or a keypad, or some other user input interface, and a monitor, such as an LCD (liquid crystal display) monitor, which may be located where the user can access them.
[0022] The controller device may be configured to facilitate, for example, the implementation of a noise removal process. The storage device may therefore include a computer program product that, when executed in the controller device (which may be a programmable or processor-based device, as noted), causes the processor-based device to perform operations that facilitate the implementation of the procedures and operations described herein. The controller device may further include peripheral devices that enable input / output functionality. Such peripheral devices may include, for example, a flash drive (e.g., a removable flash drive), or a network connection (e.g., implemented using a USB port and / or a wireless transceiver) for downloading relevant content to a connected system. Such peripheral devices may also be utilized to download software that includes computer instructions that enable the general operation of the respective system / device. Alternatively and / or in addition, in some embodiments, dedicated logic circuits, such as FPGAs (field programmable gate arrays), ASICs (application specific integrated circuits), DSP processors, graphic processing units (GPUs), accelerated processing units (APUs), application processing units, etc., may be utilized in the implementation of the controller device. Other modules that may be included with the controller device may include a user interface for providing or receiving input and output data. Additionally, in some embodiments, sensor devices such as microphones, light capture devices (e.g., CMOS-based or CCD-based camera devices), other types of optical or electromagnetic sensors, sensors for measuring environmental conditions, etc. may be coupled to the controller device and configured to observe or measure signals or data that are processed. The controller device may include an operating system.
[0023] A computer program (also known as a program, software, software application or code) comprises machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the term "machine-readable medium" refers to any non-transitory computer program product, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a non-transitory machine-readable medium that receives the machine instructions as a machine-readable signal.
[0024] In some embodiments, any suitable computer readable medium can be utilized to store instructions for performing the processes / operations / procedures described herein. For example, in some embodiments, the computer readable medium can be transitory or non-transitory. For example, non-transitory computer readable medium can include media such as magnetic media (such as hard disks, floppy disks, etc.), optical media (such as compact disks, digital video disks, Blu-ray disks, etc.), semiconductor media (such as flash memory, Electrically Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), etc.), any suitable medium that is not instantaneous or lacks any permanent semblance during transmission, and / or any suitable tangible medium. As another example, transitory computer readable medium can include networks, wires, conductors, optical fibers, circuits, any suitable medium that is not instantaneous and lacks any permanent semblance during transmission, and / or signals in suitable non-tangible media.
[0025] The subject matter of the present disclosure is further described in the accompanying materials. Although certain embodiments have been disclosed in detail herein, this is done for illustrative purposes only and is not intended to be limiting with respect to the scope of the following appended claims. Features of the disclosed embodiments can be combined, rearranged, etc., within the scope of the invention to produce further embodiments. Any other aspects, advantages, and modifications are considered to be within the scope of the claims that follow. The claims presented represent at least some of the embodiments and features disclosed herein. Other unclaimed embodiments and features are also contemplated.
[0026] In this embodiment (Listening to Silence for Speech Denoising), we introduce a deep learning model for speech denoising, a long-standing challenge in audio analysis that arises in many applications. Our approach is based on an important observation of human speech: there are often short pauses between each sentence or word. In recorded speech signals, these pauses introduce a series of periods where only noise is present. We exploit these incidental silent intervals to train a model for automatic speech denoising given only mono-channel audio. The detected silent intervals over time expose time-varying features rather than pure noise alone, allowing the model to learn the noise dynamics and suppress it from the speech signal. Experiments on multiple datasets confirm the crucial role of silent interval detection for speech denoising, and our method outperforms several state-of-the-art denoising methods, including those that accept only audio input (like our method) and those that base denoising on audiovisual input (hence requiring more information). We also show that our method enjoys superior generative properties, such as speech noise removal, that are not seen during training.
[0027] (1 Introduction) Noise is everywhere. When we listen to someone talking, the audio signal we receive is never pure and clean, but is always polluted by all kinds of noises - cars passing by, air conditioning fans revving, dogs barking, music from loudspeakers, etc. To a large extent, individuals engaged in conversation can effortlessly filter these noises (Ref. 40). In the same vein, numerous applications ranging from cellular communications to human-robot interaction rely on voice noise reduction algorithms as a fundamental building block.
[0028] Despite its critical importance, algorithmic speech denoising remains a major challenge. Given an input audio signal, speech denoising aims to separate the foreground (speech) signal from its additive background noise. This separation problem is inherently ill-posed. Classical techniques such as spectral subtraction (7, 91, 6, 66, 73) and Wiener filtering (74, 38) perform audio denoising in the spectral domain, and they are typically limited to stationary or quasi-stationary noise. In recent years, the evolution of deep neural networks has also inspired their use in audio denoising. Although superior to classical denoising techniques, existing neural network-based techniques make use of network structures deployed for general audio processing tasks (51, 83, 93) or borrowed from other domains such as computer vision (29, 24, 3, 34, 30) and generative adversarial networks (64, 65). Nevertheless, beyond reusing well-developed network models as black boxes, a fundamental question remains: what natural structures in speech can we exploit to shape the configuration of networks for better performance in speech denoising?
[0029] 1.1 Key Insight: Temporal Distribution of Silence Intervals Motivated by this question, we revisit one of the most widely implemented methods of audio denoising: spectral subtraction (7, 91, 6, 66, 73). As implemented in many commercial software such as Adobe Audition (37), this classic method requires the user to specify time intervals during which there is no foreground signal. We call such intervals silent intervals. Silence intervals are time windows that expose pure noise. The algorithm then learns noise characteristics from the silent intervals, which are then utilized to suppress additive noise in the entire input signal (through spectral domain subtraction).
[0030] Figure 2: Silent intervals over time (Top) A speech signal has many natural interruptions. In the absence of any noise, these interruptions would be presented as silent intervals (highlighted in red). (Bottom) However, most speech signals are corrupted by noise. Even light noise can overwhelm silent intervals, making them difficult to detect. If detected robustly, silent intervals can help define the noise profile over time.
[0031] Submitted to the 34th Conference on Neural Information Processing Systems (NeurIPS 2020). Do not distribute. Furthermore, the spectral subtraction method suffers from two major drawbacks: i) it requires the user's specification of silent intervals, i.e., it is not fully automatic; ii) although it is undemanding to the user, a single silent interval is not sufficient in the presence of non-stationary noise - e.g., background music. Ubiquitous, non-stationary noise in everyday life has time-variable spectral characteristics. A single silent interval only reveals the noise's spectral characteristics in that specific time span and is therefore inadequate for denoising the entire input signal. The continuity of the spectral subtraction is central to the concept of silent intervals; it is also its drawback.
[0032] In this embodiment, we introduce a deep network for speech denoising that tightly integrates silent intervals, thereby overcoming many of the limitations of classical approaches. Our goal is not to only identify a single silent interval, but to find as many possible silent intervals over time as possible. In fact, silent intervals in speech appear to be abundant: psycholinguistic studies show that there is almost always a pause after each sentence, and even after each word in an utterance (References 72, 21). Each pause, no matter how short, provides a silent interval-specific noise feature that is local in time. Taken together, these silent intervals incorporate a time-varying image of the background noise, allowing neural networks to better denoise speech signals, even in the presence of non-stationary noise (see FIG. 2).
[0033] Briefly, to interleave neural networks with established noise removal pipelines, we propose a network architecture that consists of three main components (see Fig. 1): i) one dedicated to silent interval detection, ii) another aiming to estimate the full noise from that manifested in the silent intervals, similar to the inpainting process in computer vision (Ref. 36), and iii) yet another one that cleans up the input signal.
[0034] Summary of results. Our neural network-based denoising model accepts a single channel of an audio signal and outputs a cleaned-up signal. Unlike some of the current denoising methods that take as input audiovisual signals (i.e., both audio and video), our method is applicable in a wider range of scenarios (e.g., cellular communications). We have performed extensive experiments, including ablation studies that show the effectiveness of our network components, and comparisons with several state-of-the-art denoising methods. We also evaluate our method under various signal-to-noise ratios, even under strong noise levels that we have not tested against prior methods. We show that our method consistently outperforms those methods on various denoising metrics, including those that accept only audio input (like ours) and those that denoise based on audiovisual input.
[0035] The crucial role of silent intervals in speech denoising is further confirmed by important subtle results. Even without supervision of silent interval detection, the ability to detect silent intervals is naturally generated in our network. Also, although our model is trained only on English speech, it can be easily used to denoise speech in other languages (such as Chinese, Japanese, and Korean) without additional training. Please see the supplementary material to hear our denoising results.
[0036] (2 Related Research) Speech denoisingSpeech denoising (Ref. 48) is a fundamental problem that has been studied for decades. Spectral subtraction (Refs. 7, 91, 6, 66, 73) estimates the spectrum of a clean signal by subtracting an estimate of the noise spectrum from the noisy speech spectrum. This classical method is followed by spectrogram factorization methods (Ref. 78). Wiener filtering (Refs. 74, 38) derives an enhanced signal by optimizing the mean squared error. Other methods exploit breaks in the speech, forming segments of low acoustic energy, where noise statistics can be more accurately measured (Refs. 13, 52, 79, 15, 69, 10, 11). Statistical model-based methods (Refs. 14, 32) and subspace algorithms (Refs. 12, 16) have also been explored.
[0037] The application of neural networks to audio denoising dates back to the 80s (81, 63). With the increase in computational power, deep neural networks are often used (97, 99, 98, 42). Long short-term memory networks (LSTMs) (33) are able to preserve temporal contextual information of the audio signal (47), leading to powerful results (51, 83, 93). Leveraging generative adversarial networks (GANs) (31), methods such as (64, 65) have adopted GANs in the audio domain and achieved strong performance.
[0038] Audio signal processing methods operate on the raw waveform or spectrogram by short-time Fourier transform (STFT). Some operate directly on the waveform (22, 62, 54, 50), others use Wavenet (84) for audio noise reduction (68, 70, 28). Many other methods, such as (49, 87, 56, 92, 41, 100, 9), study the spectrogram of the audio signal, which contains both magnitude and phase information. There are works that discuss how to use the spectrogram to its full potential (86, 61), but one of the drawbacks is that an inverse STFT needs to be applied. In response to this, there are also works that investigate how to overcome artifacts from time aliasing (46, 27, 26, 88, 19, 94, 55).
[0039] Speech denoising has also been explored in conjunction with computer vision due to the association between speech and facial features (Reference 8). Methods such as (References 29, 24, 3, 34, 30) use various network structures to enhance the audio signal to the best of their ability. Adeel et al. (Reference 1) even uses lip reading to filter background noise in speech.
[0040] Deep Learning in Other Audio Processing Tasks. Inspired by computer vision, deep learning has been widely used for lip reading, speech recognition, speech separation, and many audio processing or audio-related tasks (58, 60, 5, 4). Methods such as (45, 17, 59) are able to reconstruct speech from pure facial features. Methods such as (2, 57) exploit facial features to improve the accuracy of speech recognition. Speech separation is one of the areas where computer vision has been put to great use. Methods such as (23, 58, 18, 102) have achieved impressive results, allowing speech separation from a single audio signal that was not possible before. Recently, Zhang et al. (101) proposed a new operation called Harmonic Convolution, encouraging the network to derive audio priors, which has even been shown to further improve the quality of speech separation.
[0041] (3) Learning to remove audio noise We present a neural network that exploits the time distribution of silent intervals for speech denoising. The input to our model is a spectrogram of noisy speech (Refs. 96, 20, 77), which can be viewed as a 2D image of size T×F with two channels, where T represents the time length of the signal and F is the number of frequency bins. The two channels store the real and imaginary parts of the STFT, respectively. After training, the model generates another spectrogram of the same size as the suppressed noise.
[0042] First, we train our proposed network structure in an end-to-end manner with only supervision of denoising (Sec. 3.2); it already outperforms the state-of-the-art methods we compare it to. Furthermore, we incorporate supervision for silent interval detection (Sec. 3.3) and obtain even better denoising results (see Sec. 4).
[0043] (3.1 Network Structure) Classical denoising algorithms work in three general stages: identifying silent intervals, estimating noise features, and removing noise. We propose to weave learning throughout this process: we revisit each stage prompted by neural networks to form a novel speech denoising model. Because we can jointly connect these networks to estimate gradients, we can efficiently train the model on a large scale of audio data. Figure 1 shows this model, which we describe below.
[0044] Silence interval detection. The first component is dedicated to detecting silent intervals in the input signal. The input to this component is the spectrogram of the input (noisy) signal x. The spectrogram S x is first encoded into a 2D feature map by a 2D convolutional encoder, which is then processed by a bidirectional LSTM (33, 75), followed by two fully connected (FC) layers (see network details in A below). The bidirectional LSTM is suitable for processing the time series features resulting from the spectrogram (53, 39, 67, 18), and the FC layers apply features to each time sample to adapt to the variable length of the input. The output from this network component is a vector D(S x ). D(S x Each element of,x,is a scalar in [0,1] (after application of a sigmoid function) indicating the confidence score of,small time segments that are silent. We choose each time segment to have 1 / 30,seconds, small enough to capture short speech interruptions,,and large enough to allow robust prediction (see Sect. 3.3).
[0045] Figure 3: Examples of intermediate and final results: (a) Spectrogram of the noisy input signal, (b) superposition of the clean speech signal and (c) noise. The black areas in (b) indicate the ground truth silent intervals. (d) Noise exposed to automatically appearing silent intervals, i.e., the output of the silent interval detection component when the whole network is trained without silent interval supervision (recall from section 3.2). (e) Noise exposed to detected silent intervals, i.e., the output of the silent interval detection component when the network is trained without silent interval supervision (recall from section 3.3). (f) Estimated noise profile using sub-sections (a) and (e) of Figure 3 as input to the noise estimation component. (g) Output of the final denoised spectrogram.
[0046] The output vector D(S x ) is then expanded into a longer mask that we denote as m(x), where each element of this mask indicates the confidence with which to classify each sample of the input signal x as pure noise (see FIG. 3(e)). With this mask, the
number
number
[0047] Noise estimation. Resulting from silent interval detection
number
[0048] The inputs to this component are a noisy audio signal and
number
number
[36] . To this end, we decompose the image into two feature maps by two individual 2D convolutional encoders.
number
number
[0049] Remove noise. Finally, we clean up the noise from the input signal x. We take as input the input speech spectrogram S x, and the estimated full noise spectrogram
number
number
[0050] In the final step, we compute the input speech spectrogram S x and mask
number
number
number
[0051] (3.2 Loss Functions and Training) Since subgradients exist at every stage, we can train our network in an end-to-end manner with stochastic gradient descent. We optimize the following loss function:
number
number
number
[0052] Spontaneous occurrence of silent intervals. While producing plausible denoising results (see Sect. 4.4), the end-to-end training process is unsupervised for silence interval detection: the loss function (1) is solely responsible for recovering the noise and clean speech signals. However, somewhat surprisingly, the ability to detect silence intervals is automatically produced as the output of the first network component.
number
[0053] (3.3 Silence interval monitoring) As the model is learning to detect its own silence intervals, we can directly supervise the silence interval detection and further improve the quality of the noise removal. Our first attempt was to add a term in (1) that penalizes the discrepancy between the detected silence interval and its ground truth. However, our experiments show that this method is not effective (see Sect. 4.4). Instead, we trained our network in two successive stages.
[0054] First, we trained the silence interval detection component through the following loss function:
number
number
number
number
[0055] Next, we train the noise estimation and removal component with loss function (1). This training phase starts by ignoring the silence detection component. In loss function (1),
number
number
[0056] (4 Experiments) This section presents the primary evaluation of our method, a comparison with several baselines and prior work, and an ablation study. We also refer the reader to supplementary material for a full description of our network architecture, implementation details, additional evaluations, and audio examples (including the effects of audio organized in supplementary documents and offline web pages).
[0057] 4.1 Experimental Setup Dataset Construction. To construct the training and test data, we exploited a publicly available audio dataset. We used AVSPEECH to obtain a clean speech signal (ref. 18), from which we randomly selected 2448 videos (total length 4.5 hours) and extracted their speech audio channel. Between them, we used 2214 training videos and 234 test videos, so that the training and test speech are well separated. All these speech videos are in English and were purposefully chosen: as we show in the supplementary material, our model trained on this dataset allows for easy speech denoising in other languages.
[0058] We use two datasets as background noise: DEMAND (Reference 82) and Google's AudioSet (Reference 25). Both consist of environmental noise, traffic noise, music, and many other types of noise. DEMAND has been used in prior denoising studies (e.g., References 64, 28, 83). Furthermore, AudioSet is much larger and more diverse than DEMAND, and therefore more challenging when used as noise. Figure 4 shows some examples of noise. Our evaluation is performed on both datasets separately.
[0059] Figure 4: Noise Gallery. We show four examples of noise from the noise dataset. Noise 1) is stationary (white) noise, the other three are not. Noise 2) is a monologue during a meeting. Noise 3) is party noise from individuals talking and laughing, with background noise. Noise 4) is noise on the street from individuals shouting and yelling, with additional traffic noise such as vehicles driving and honking.
[0060] Due to the linearity of acoustic wave propagation, we can superimpose a clean speech signal onto the noise and synchronize the noisy input signal (as in previous studies (References 64, 28, 83)). When synchronizing the noisy input signal, we randomly selected the signal-to-noise ratio (SNR) from seven distinct values: -10 dB, -7 dB, -3 dB, 0 dB, 3 dB, 7 dB, and 10 dB; by mixing the foreground speech with appropriately measured noise, we generated a noisy signal with the selected SNR. For example, an SNR of -10 dB means that the noise is 10 times more powerful than the speech (see Figure 7). The range of SNRs in our evaluation (i.e., [-10 dB, 10 dB]) is significantly larger than those tested in previous studies.
[0061] To supervise our silent interval detection (recall Section 3.3), we need ground truth labels of silent intervals. To this end, we divide each clean speech signal into time segments, each lasting 1 / 30th of a second. We classify a time segment as silent when the total acoustic energy of that segment falls below a threshold. Because the speech is clean, this automatic classification process is robust.
[0062] Comparison of methods. We compare our method with several existing methods also designed for speech denoising, including both classical and recently proposed learning-based methods, which we refer to as: i) ours, our model trained on monitoring silent intervals (recall Section 3.3); ii) baseline thresholding, a baseline method that utilizes a threshold of acoustic energy to classify silent intervals (same as our automatic labelling method in Section 4.1, but applied to noisy input signals) and then our trained noise estimate and denoising network for speech denoising. iii) our GTSI, another reference method that utilizes our trained noise estimation and removal network but hypothetically uses ground truth silence intervals; iv) spectral gating, a classical speech denoising algorithm based on spectral subtraction (73); v) Adobe Audition (37), one of the most widely used professional audio processing software, we utilize its machine learning based noise reduction features provided in the latest Adobe Audition CC 2020 with default parameters to batch process all our test data; vi) SEGAN (64), one of the most recent audio-only speech enhancement methods based on generative adversarial networks; vii) DFL (28), a recently proposed speech denoising method based on a loss function over deep network features; 1 viii) VSE (24), a learning-based method that takes both video and audio as input and leverages both the audio signal and mouth movements (from video footage) for speech denoising. We cannot compare with another audiovisual method (Ref. 18) because no source code or executable files have been made publicly available.
[0063] For a fair comparison, we use the same dataset to train all methods (except for the learning base and the spectral gating, which is not Adobe Audition commercially shipped as a black box). For SEGAN, DFL, and VSE, we use their source code published by the authors. The audiovisual denoising method VSE also requires video footage, which is available in AVSPEECH.
[0064] (4.2 Evaluation of Audio Noise Reduction) metric. Due to the perceptual nature of audio processing tasks, there is no single, widely accepted metric for quantitative evaluation and comparison. We therefore evaluate our method on six different metrics, all of which are frequently used to evaluate the quality of audio processing: i) Perceptual Evaluation of Speech Quality (PESQ) (Reference 71), ii) Segmental Signal-to-Noise Ratio (SSNR) (Reference 76), iii) Short-Term Objective Intelligibility (STOI) (Reference 80), iv) Mean Opinion Score (MOS) Predictor of Signal Distortion (CSIG) (Reference 35), v) MOS Predictor of Background Noise Intrusiveness (CBAK) (Reference 35), and vi) MOS Predictor of Overall Signal Quality (COVL) (Reference 35).
[0065] Figure 5: Quantitative comparison. We measure the quality of denoising with six metrics (corresponding columns). Comparisons are performed using noise from DEMAND and AudioSet separately. Our GTSI (black) uses ground truth silence intervals. Although not a practical approach, it serves as an upper bound reference for all methods.
[0066] Figure 6: Quality of denoising with respect to the input SNR. The denoising results measured by PESQ for each method associated with different input SNRs. The results measured by other metrics are shown in Fig. 8.
[0067] result. We train two separation models using the DEMAND and AudioSet noise datasets, respectively, and compare them with other models trained on the same datasets. We evaluate the average metric values and report them in Figure 5. Across all metrics, our method consistently outperforms the others.
[0068] We break down the performance of each method with respect to SNR levels from -10 dB to 10 dB for both noise datasets. Results are reported in Fig. 6 for PESQ (see Fig. 8). None of the previous studies we compared reported results at these low SNR levels (<0 dBs). Nevertheless, across all levels of input SNR, our method performed best, indicating that our approach is fairly robust to both light and extreme noise.
[0069] It is worth noting from Figure 6 that our GTSI method performs even better. Recall that this is our model, but it is outfitted with ground truth silence intervals. Although not practical (due to the need for ground truth silence intervals), our GTSI confirms the importance of silence intervals for noise reduction: quality silence interval detection helps improve the quality of speech denoising.
[0070] (4.3 Evaluation of Silence Interval Detection) Due to the importance of silence intervals for speech noise removal, we also evaluate the quality of our silence interval detection and compare two alternatives: a baseline threshold and a speech detector (VAD) (Reference 95). The former is described above, while the latter classifies each time window of the audio signal as either accompanied by human voice or not (References 43, 44). We use an off-the-shelf VAD developed by Google's WebRTC project and reported to be one of the most available.
[0071] We evaluate these methods using four standard statistical metrics: precision, recall, F1 score, and accuracy. We follow the standard definitions of these metrics summarized in C.1. These metrics are based on the definition of positive / negative conditions, where a positive condition indicates a time segment that has been classified as a silence segment, and a negative condition indicates a non-silence classification. Thus, the higher the metric, the better the detection technique.
[0072] Table 1 shows that under all metrics, our method consistently outperforms the alternatives. Between VAD and the baseline threshold, VAD has higher precision and lower recall, which means that VAD is overly conservative and the baseline threshold is overly aggressive when detecting silent intervals (see FIG. 9). Our method achieves a better balance and therefore detects silent intervals more accurately.
[0073] [Table 1] [Table 1] Table 1: Results of silent interval detection. The metrics are measured using our test signals with SNRs ranging from -10 dB to 10 dB. The definitions of these metrics are summarized in Section C.1 below.
[0074] [Table 2] [Table 2] Table 2: Ablation study. We vary the network components and training losses and evaluate the quality of denoising on different metrics. Our proposed approach performs the best.
[0075] (4.4 Ablation Study) In addition, we perform a series of ablation studies to understand the effectiveness of individual network components and loss terms (see D.1 below for further details). In Table 2, "Our W / O SID Loss" refers to the training method presented in Section 3.2 (i.e., without monitoring of silent intervals). "Our Joint Loss" refers to the end-to-end training approach that optimizes loss function (1) with additional term (2). Also, "Our w / o NE Loss" utilizes our two-stage training (Section 3.3) but without the loss term of noise estimation - i.e., without the first term of (1). Comparing these alternative training approaches, our two-stage training with monitoring of silent intervals (referred to as "ours") performs best. We also note that our W / O SID loss - i.e., without supervision - is already superior to the methods we compared in Fig. 5 and that ours further improves the quality of noise removal. This indicates the effectiveness of our proposed training method for silent interval detection.
[0076] We also experimented with two variations of our network structure: the first, called "our w / o SID comp", turns off silent interval detection: the silent interval detection component always outputs an all-zero vector. The second one, labeled "our w / o NR comp," utilizes simple spectral subtraction to replace our noise-removing component. Table 2 shows that under all tested metrics, both variants perform worse than our method, demonstrating the effectiveness of our proposed network structure.
[0077] Furthermore, we have investigated to what extent the accuracy of silent interval detection affects the quality of speech denoising. We show that as silent interval detection becomes less accurate, the quality of denoising decreases. Presented in detail in the following D.2, these experiments reinforce our intuition that silent intervals are instructive for speech denoising tasks.
[0078] (5 conclusions) Speech denoising has been a long-standing challenge. We present a new network structure that exploits the abundance of silent intervals in speech. Even without monitoring of silent intervals, our network is able to plausibly denoise speech signals, whereas the ability to detect silent intervals is automatically generated. We reinforce this ability. Our explicit monitoring of silent intervals allows the network to detect them more accurately, thereby further improving the performance of speech denoising. As a result, under various denoising metrics, our method consistently outperforms several state-of-the-art audio denoising models.
[0079] (Wider Impact) Good quality speech noise reduction is desirable in many applications: human-robot interaction, cellular communications, hearing aids, teleconferencing, music recording, film production, news reporting, and surveillance systems, etc. Thus, we hope that our proposed noise reduction method will find impact in these applications, whether in real-world systems or as a cornerstone for future technologies.
[0080] In our experiments, we train our model using only English speech to demonstrate its generalized property - its ability to denoise speech beyond English. It is intended that we demonstrate denoising for Japanese, Chinese, and Korean speech: they are linguistically and phonologically distant from English (as opposed to other English "siblings" such as German and Dutch). Still, our model may be biased in favor of speech languages and cultures that are closer to English or contain frequent pauses that manifest silent intervals. A deeper understanding of this potential bias requires future investigations in parallel with linguistic and socio-cultural insights.
[0081] Finally, it is natural to extend our model to denoise general audio signals or even signals beyond audio (such as denoising gravitational waves (Ref. 90)). If successful, our model could have even wider impact. However, pursuing this extension requires a sensible definition of a "silent interval". Ultimately, the concept of "noise" in the general context of signal processing depends on the specific application: noise in one application may be of another signal. In order to train a neural network that utilizes the general notion of silent intervals, care must be taken to avoid biasing it towards specific types of noise.
[0082] (References) (Reference 1) A. Adeel, M. Gogate, A. Hussain, and W. M. Whitmer. Lip-reading driven deep learning approach for speech enhancement. IEEE Transactions on Emerging Topics in Computational Intelligence, page 1-10, 2019. ISSN 2471-285x. doi: 10.1109 / tetci.2019.2917039. URL http: / / dX.doi.org / 10.1109 / tetci.2019.2917039. (Reference 2) T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2018. (Reference 3) T. Afouras, J. S. Chung, and A. Zisserman. The conversation: Deep audio-visual speech enhancement. In Proc. Interspeech 2018, pages 3244-3248, 2018. doi: 10.21437 / Interspeech. 2018- 1400. URL http: / / dx.doi.org / 10.21437 / Interspeech.2018-1400. (Reference 4) R. Arandjelovic and A. Zisserman. Objects that sound. In Proceedings of the European Conference on Computer Vision (ECCV), pages 435-451, 2018. (Reference 5) Y. Aytar, C. Vondrick, and A. Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in neural information processing systems, pages 892-900, 2016. (Reference 6) M. Berouti, R. Schwartz, and J. Makhoul. Enhancement of speech corrupted by acoustic noise. In ICASSP 79. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 4, pages 208-211, 1979. (Reference 7) S. Boll. Suppression of acoustic noise in speech using spectral subtraction. IEEE Transactions on Acoustics, Speech, and Signal Processing, 27(2):113-120, 1979. (Reference 8) C. Busso and S. S. Narayanan. Interrelation between speech and facial gestures in emotional utterances: A single subject study. IEEE Transactions on Audio, Speech, and Language Processing, 15(8):2331-2347, 2007. (Reference 9) J Chen and D. Wang. Long short-term memory for speaker generalization in supervised speech separation. Acoustical Society of America Journal, 141(6):4705-4714, June 2017. doi: 10.1121 / 1.4986931. (Reference 10) I. Cohen. Noise spectrum estimation in adverse environments: improved minima controlled recursive averaging. IEEE Transactions on Speech and Audio Processing, 11(5):466-475, 2003. (Reference 11) I. Cohen and B. Berdugo. Noise estimation by minima controlled recursive averaging for robust speech enhancement. IEEE Signal Processing Letters, 9(1): 12-15, 2002. (Reference 12) M. Dendrinos, S. Bakamidis, and G. Carayannis. Speech enhancement from noise: A regener-ative approach. Speech Commun., 10(1):45-67, Feb. 1991. ISSN 0167-6393. doi: 10.1016 / 0167-6393(91)90027-q. URL https: / / doi.org / 10.1016 / 0167-6393(91)90027-0. (Reference 13) G. Doblinger. Computationally efficient speech enhancement by spectral minima tracking in subbands. In in Proc. Eurospeech, pages 1513-1516, 1995. (Reference 14) Y. Ephraim. Statistical-model-based speech enhancement systems. Proceedings of the IEEE, 80(10):1526-1555, 1992. (Reference 15) Y. Ephraim and D. Malah. Speech enhancement using a minimum mean-square error log-spectral amplitude estimator. IEEE Transactions on Acoustics, Speech, and Signal Processing, 33(2):443-445, 1985. (Reference 16) Y. Ephraim and HL Van Trees. A signal subspace approach for speech enhancement. IEEE Transactions on Speech and Audio Processing, 3(4):251-266, 1995. (Reference 17) A. Ephrat, T. Halperin, and S. Peleg. Improved speech reconstruction from silent video. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pages 455-462, 2017. (Reference 18) A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. ACM Transactions on Graphics, 37(4):1-11, July 2018. ISSN 0730-0301. doi: 10.1145 / 3197517.3201357. URL http: / / dx.doi.org / 1O.1145 / 3197517.3201357. (Reference 19) H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux. Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 708-712, 2015. (Reference 20) J. L. Flanagan. Speech Analysis Synthesis and Perception. Springer-Verlag, 2nd edition, 1972. ISBN 9783662015629. (Reference 21) K. L. Fors. Production and perception of pauses in speech. PhD thesis, Department of Philosophy, Linguistics, and Theory of Science, University of Gothenburg, 2015. (Reference 22) S.-W. Fu, Y. Tsao, X. Lu, and H. Kawai. Raw waveform-based speech enhancement by fully convolutional networks. 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Dec. 2017. doi: 10.1109 / apsipa.2017.8281993. URL http: / / dX.doi.org / 10.1109 / APSIPA.2017.8281993. (Reference 23) A. Gabbay, A. Ephrat, T. Halperin, and S. Peleg. Seeing through noise: Visually driven speaker separation and enhancement, 2017. (Reference 24) A. Gabbay, A. Shamir, and S. Peleg. Visual speech enhancement, 2017. (Reference 25) J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 2017, New Orleans, LA, 2017. (Reference 26) T. Gerkmann, M. Krawczyk-Becker, and J. Le Roux. Phase processing for single-channel speech enhancement: History and recent advances. IEEE Signal Processing Magazine, 32(2): 55-66, 2015. (Reference 27) F. G. Germain, G. J. Mysore, and T. Fujioka. Equalization matching of speech recordings in real-world environments. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 609-613, 2016. (Reference 28) F. G. Germain, Q. Chen, and V. Koltun. Speech denoising with deep feature losses. In Proc. Interspeech 2019, pages 2723-2727, 2019. doi: 10.21437 / Interspeech.2019-1924. URL http: / / dx.doi.org / 10.21437 / Interspeech.2019-1924. (Reference 29) L. Girin, J.-L. Schwartz, and G. Feng. Audio-visual enhancement of speech in noise. The Journal of the Acoustical Society of America, 109(6):3007-3020, 2001. doi: 10.1121 / 1. 1358887. URL https: / / doi.org / 10.1121 / 1.1358887. (Reference 30) M. Gogate, A. Adeel, K. Dashtipour, P. Derleth, and A. Hussain. Av speech enhancement challenge using a real noisy corpus, 2019. (Reference 31) I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Proceedings ofthe 27th International Conference on Neural Information Processing Systems - Volume 2, Nips' 14, page 2672-2680, Cambridge, MA, USA, 2014. MIT Press.(Reference 32) H.-G. Hirsch and C. Ehrlicher. Noise estimation techniques for robust speech recognition. 1995 International Conference on Acoustics, Speech, and Signal Processing, 1:153-156 vol.1, 1995 . (Reference 33) S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9:1735-80, 12 1997. doi: 10.1162 / neco.1997.9.8.1735. (Table 34)J.-C. Hou, S.-S. Wang, Y.-H. Lai, Y. Tsao, H.-W. Chang, and H.-m. Wang. Audio-visual speech enhancement using multimodal deep convolutional neural networks. IEEE Transactions on Emerging Topics in Computational Intelligence, 2, 03 2018. doi:10.1109 / tetci.2017.2784878. (Page 35)Y. Hu and P. Loizou. Evaluation of objective quality measures for speech enhancement. Audio, Speech, and Language Processing, IEEE Transactions on, 16:229-238, 02 2008. doi:10.1109 / tasl.2007.911054. (Picture No. 36)S. Iizuka , E. Simon-Serra , and H. Ishikawa . Globally and locally consistent image completion. ACM Trans. Graph., 36(4), July 2017. ISSN 0730-0301. doi: 10.1145 / 3072959.3073659. URL https: / / doi.org / 10.1145 / 3072959.3073659. (Part 37)A. Inc. Adobe Audition, 2020. URL https: / / www.adobe.com / products / audition.html. (Reference 38) Jae Lim and A. Oppenheim. All-pole modeling of degraded speech. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(3):197-210, 1978. (Reference 39) N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu. Efficient neural audio synthesis, 2018. (Reference 40) AJE Kell and JH McDermott. Invariance to background noise as a signature of non-primary auditory cortex. Nature Communications, 10(1):3958, Sept. 2019. ISSN 2041-1723. doi: 10.1038 / s41467-019-11710-y. URL https: / / doi.org / 10.1038 / s41467-019-11710-y. (Reference 41) A. Kumar and D. Florencio. Speech enhancement in multiple-noise conditions using deep neural networks. Interspeech 2016, Sept. 2016. doi: 10.21437 / interspeech.2016-88. URL http: / / dx.doi.org / 10.21437 / Interspeech.2016-88. (Reference 42) A. Kumar and D. A. F. Florencio. Speech enhancement in multiple-noise conditions using deep neural networks. In Interspeech, 2016. (Reference 43) R. Le Bouquin Jeannes and G. Faucon. Proposal of a voice activity detector for noise reduction. Electronics Letters, 30(12):930-932, 1994. (Reference 44) R. Le Bouquin Jeannes and G. Faucon. Study of a voice activity detector and its influence on a noise reduction system. Speech Communication, 16(3):245-254, 1995. ISSN 0167-6393. doi: https: / / doi.org / 10.1016 / 0167-6393(94)00056-G. URL http: / / www.sciencedirect.com / science / article / pii / 016763939400056G. (Reference 45) T. Le Cornu and B. Milner. Generating intelligible audio speech from visual speech. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 25(9): 1751-1761, 2017. (Reference 46) J. Le Roux and E. Vincent. Consistent wiener filtering for audio source separation. IEEE Signal Processing Letters, 20(3):217-220, 2013. (Table 47)ZC Lipton, J. Berkowitz, and C. Elkan. A Critical Review of Recurrent Neural Networks for Sequence Learning , 2015 . (Page 48)PC Loizou. Speech Enhancement: Theory and Practice. CRC Press, Inc., USA, 2nd edition, 2013. ISBN 1466504218. (Page 49)X. Lu, Y. Tsao, S. Matsuda, and C. Hori. Speech enhancement based on deep denoising autoencoder. In Interspeech , 2013 . (Page 50)Y. Luo and N. Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE / ACM Trans. Audio, Speech and Language. Proc., 27(8):1256-1266, Aug. 1999; 2019. ISSN 2329–9290. doi: 10.1109 / taslp.2019.2915167. URL https: / / doi.org / 10.1109 / TASLP. (Additional 51) Maas AL, Le QV, O'Neil TM, Vinyals O, Nguyen P, and Ng AY. Recurrent neural networks for noise reduction in robust asr. In Interspeech , 2012 . (Reference 52) R. Martin. Noise power spectral density estimation based on optimal smoothing and minimum statistics. IEEE Transactions on Speech and Audio Processing, 9(5):504-512, 2001. (Reference 53) S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio. Samplernn: An unconditional end-to-end neural audio generation model, 2016. (Reference 54) M. Michelashvili and L. Wolf. Audio denoising with deep network priors, 2019. (Reference 55) J. A. Moorer. A note on the implementation of audio processing by short-term fourier transform. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 156-159, 2017. (Reference 56) A. Narayanan and D. Wang. Ideal ratio mask estimation using deep neural networks for robust speech recognition. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 7092-7096, 2013. (Picture No. 57)K. Noda , Y. Yamaguchi , K. Nakadai , HG Okuno , and T. Ogata . Audio-visual speech recognition using deep learning. Applied Intelligence, 42(4):722-737, June 2015. ISSN 0924-669x. doi: 10.1007 / s10489-014-0629-7. URL https: / / doi.org / 10. 1007 / s10489-014-0629-7 (Part 58)A. Owens and AA Efros. Audio-visual scene analysis with self-supervised multisensory features. Lecture Notes in Computer Science, pages 639-658, 2018. ISSN 1611-3349. doi: 10.1007 / 978-3-030-01231-1¥_39. URL http: / / dX.doi.org / 10.1007 / 978-3-030-01231-1_39. (Part 59)A. Owens , P. Isola , J. McDermott , A. Torralba , EH Adelson , and WT Freeman . Visually indicated sounds. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. doi:10.1109 / cvpr.2016.264. URL http: / / dx.doi.org / 10.1109 / CVPR.2016. 264. (Reference 60) A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba. Ambient sound provides supervision for visual learning. In European conference on computer vision, pages 801-816. Springer, 2016. (Reference 61) K. Paliwal, K. Wojcicki, and B. Shannon. The importance of phase in speech enhancement. Speech Commun., 53(4):465-494, Apr. 2011. ISSN 0167-6393. doi: 10.1016 / j.specom.2010. 12.003. URL https: / / doi.org / 10.1016 / j.specom.2010.12.003. (Reference 62) A. Pandey and D. Wang. A new framework for supervised speech enhancement in the time domain. In Proc. Interspeech 2018, pages 1136-1140, 2018. doi: 10.21437 / Interspeech. 2018-1223. URL http: / / dx.doi.org / 10.21437 / Interspeech .2018-1223. (Reference 63) S. Parveen and P. Green. Speech enhancement with missing data techniques using recurrent neural networks. In 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages I-733, 2004. (Reference 64) S. Pascual, A. Bonafonte, and J. Serra. Segan: Speech enhancement generative adversarial network. In Proc. Interspeech 2017, pages 3642-3646, 2017. doi: 10.21437 / Interspeech. 2017- 1428. URL http: / / dx.doi.org / 10.21437 / Interspeech .2017-1428. (Reference 65) S. Pascual, J. Serra, and A. Bonafonte. Towards generalized speech enhancement with generative adversarial networks. In Proc. Interspeech 2019, pages 1791-1795, 2019. doi: 10.21437 / Interspeech.2019-2688. URL http: / / dx.doi.org / 10. 21437 / Interspeech. 2019-2688. (Reference 66) L. ping Yang and Q.-J. Fu. Spectral subtraction-based speech enhancement for cochlear implant patients in background noise. The Journal ofthe Acoustical Society ofAmerica, 117 3 Pt 1:1001-4, 2005. (Reference 67) H. Purwins, B. Li, T. Virtanen, J. Schluter, S.-Y. Chang, and T. Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing, 13(2): 206-219, May 2019. ISSN 1941-0484. doi: 10.1109 / jstsp.2019.2908700. URL http: / / dX.doi.org / 10.1109 / JSTSP.2019.2908700. (Reference 68) K. Qian, Y. Zhang, S. Chang, X. Yang, D. Florencio, and M. Hasegawa-Johnson. Speech enhancement using bayesian wavenet. In Proc. Interspeech 201 7, pages 2013-2017, 2017. doi: 10.21437 / Interspeech.2017- 1672. URL http: / / dx.doi.org / 10.21437 / Interspeech.2017-1672. (Reference 69) S. Rangachari, P. C. Loizou, and Yi Hu. A noise estimation algorithm with rapid adaptation for highly nonstationary environments. In 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages I-305, 2004. (Reference 70) D. Rethage, J. Pons, and X. Serra. A wavenet for speech denoising. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5069-5073, 2018. (Reference 71) A. Rix, J. Beerends, M. Hollier, and A. Hekstra. Perceptual evaluation of speech quality (pesq): A new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01 CH37221), volume 2, pages 749-752 vol.2, 02 2001. ISBN 0-7803-7041-4. doi: 10.1109 / icassp.2001.941023. (Reference 72) S. R. Rochester. The significance of pauses in spontaneous speech. Journal of Psycholinguistic Research, 2(1):51-81, 1973. (Reference 73) T. Sainburg. Noise reduction in python using spectral gating. https: / / github.com / timsainb / noisereduce, 2019. (Reference 74) P. Scalart and J. V. Filho. Speech enhancement based on a priori signal to noise estimation. In 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, volume 2, pages 629-632 vol. 2, 1996. (Reference 75) M. Schuster and K. Paliwal. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 4532673-2681, 12 1997. doi: 10.1109 / 78.650093. (Reference 76) M. A. C. Schuyler R. Quackenbush, Thomas P. Barnwell. Objective Measures Of Speech Quality. Prentice Hall, Englewood Cliffs, NJ, 1988. ISBN 9780136290568. (Reference 77) E. Sejdic, I. Djurovic, and L. Stankovic. Quantitative performance analysis of scalogram as instantaneous frequency estimator. IEEE Transactions on Signal Processing, 56(8):3837-3845, 2008. (Reference 78) P. Smaragdis, C. Fevotte, G. J. Mysore, N. Mohammadiha, and M. Hoffman. Static and dynamic source separation using nonnegative factorizations: A unified view. IEEE Signal Processing Magazine, 31(3):66-75, 2014. (Reference 79) K. V. Sorensen and S. V. Andersen. Speech enhancement with natural sounding residual noise based on connected time-frequency speech presence regions. EURASIP J. Adv. Signal Process, (Reference 80) C. Taal, R. Hendriks, R. Heusdens, and J. Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 4214-4217, 04 2010. doi: 10.1109 / icassp. 2010.5495701. (Reference 81) S. Tamura and A. Waibel. Noise reduction using connectionist models. In ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing, pages 553-556 vol.1, 1988. (Reference 82) J. Thiemann, N. Ito, and E. Vincent. The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In 21st International Congress on Acoustics, Montreal, Canada, June 2013. Acoustical Society of America. doi: 10.5281 / zenodo.1227120. URL https : / / hal . inria . fr / ha1- 00796707. The dataset itself is archived on Zenodo, with DOI 10.5281 / zenodo.1227120. (Reference 83) C. Valentini-Botinhao, X. Wang, S. Takaki, and J. Yamagishi. Investigating rnn-based speech enhancement methods for noise-robust text-to-speech. In 9th ISCA Speech Synthesis Workshop, pages 146-152, 2016. doi: 10.21437 / ssw.2016-24. URL http : / / dx . doi . org / 10 . 21437 / SSW.2016-24. (Reference 84) A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. ArXiv, abs / 1609.03499, 2016. (Reference 85) D. Wang and J. Chen. Supervised speech separation based on deep learning: An overview. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 26(10): 1702-1726, Oct 2018. ISSN 2329-9304. doi: 10.1109 / taslp.2018.2842159. URL http: / / dx.doi.org / 10.1109 / TASLP.2018.2842159. (Reference 86) D. Wang and Jae Lim. The unimportance of phase in speech enhancement. IEEE Transactions on Acoustics, Speech, and Signal Processing, 30(4):679-681, 1982. (Reference 87) Y. Wang and D. Wang. Cocktail party processing via structured prediction. In Proceedings ofthe 25th International Conference on Neural Information Processing Systems - Volume 1, Nips' 12, page 224-232, Red Hook, NY, USA, 2012. Curran Associates Inc. (Reference 88) Y. Wang and D. Wang. A deep neural network for time-domain signal reconstruction. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4390-4394, 2015. (Reference 89) Y. Wang, A. Narayanan, and D. Wang. On training targets for supervised speech separation. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 22(12):1849-1858, 2014. (Reference 90) W. Wei and E. Huerta. Gravitational wave denoising of binary black hole mergers with deep learning. Physics Letters B, 800: 135081, 2020. (Reference 91) M. R. Weiss, E. Aschkenasy, and T. W. Parsons. Study and development of the intel tech-nique for improving speech intelligibility. Technical report nsc-fr / 4023, Nicolet Scientific Corporation, 1974. (Reference 92) F. Weninger, J. R. Hershey, J. Le Roux, and B. Schuller. Discriminatively trained recurrent neural networks for single-channel speech separation. In 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 577-581, 2014. (Reference 93) F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Roux, JR Hershey, and B. Schuller. Speech enhancement with lstm recurrent neural networks and its application to noise-robust asr. In Proceedings of the 12th International Conference on Latent Variable Analysis and Signal Separation - Volume 9237, Lva / ica 2015, page 91-99, Berlin, Heidelberg, 2015. Springer- Verlag. ISBN 9783319224817. doi: 10.1007 / 978-3-319-22482-4¥_11. URL https: / / doi.org / 10.1007 / 978-3-319-22482-4_11. (Reference 94) DS Williamson and D. Wang. Time-frequency masking in the complex domain for speech dereverberation and denoising. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 25(7):1492-1501, 2017. (Reference 95) J. Wiseman. Py-webrtcvad. https: / / github.com / wiseman / py-webrtcvad, 2019. (Reference 96) L. Wyse. Audio spectrogram representations for processing with convolutional neural networks, 2017. (Reference 97) Y. Xu, J. Du, L. Dai, and C. Lee. An experimental study on speech enhancement based on deep neural networks. IEEE Signal Processing Letters, 21(1):65-68, 2014. (Reference 98) Y. Xu, J. Du, L. Dai, and C. Lee. A regression approach to speech enhancement based on deep neural networks. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 23(1): 7-19, 2015. (Reference 99) Y. Xu, J. Du, Z. Huang, L.-R. Dai, and C.-H. Lee. Multi-objective learning and mask-based post-processing for deep neural network based speech enhancement. In Interspeech, 2015. (Reference 100) X. Zhang and D. Wang. A deep ensemble learning method for monaural speech separation. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 24(5):967-977, 2016. (Reference 101) Z. Zhang, Y. Wang, C. Gan, J. Wu, JB Tenenbaum, A. Torralba, and WT Freeman. Deep audio priors emerge from harmonic convolutional networks. In International Conference on Learning Representations, 2020. URL https : / / openreview . net / forum id=rygjHXrYDB. (Reference 102) H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), pages 570-586, 2018.
[0083] (Additional explanation, listen to silence to remove audio noise) (A: Network structure and training details) We now present the details of our network structure and training configuration.
[0084] The silence interval detection component of our model consists of a 2D convolutional layer, a bidirectional LSTM, and two FC layers. The parameters of the convolutional layers are shown in Table 3. Each convolutional layer is followed by a batch normalization layer with a ReLU activation function. The hidden magnitude of the bidirectional LSTM is 100. The two FC layers interleaved with the ReLU activation function have hidden magnitudes of 100 and 1, respectively.
[0085] [Table 3] [Table 3]
[0086] The noise estimation component of our model is fully convoluted and consists of two encoders and one decoder. The two encoders handle the noisy signal and the imperfect noise profile, respectively; they have the same configuration (shown in Table 4) but with different weights. The two feature maps resulting from the two encoders are concatenated in a per-channel manner before feeding into the decoder. In Table 4, all layers except the last one are followed by a batch normalization layer with a ReLU activation function. In addition, there are skip connections between layers 2 and 14, and between layers 4 and 12.
[0087] [Table 4] [Table 4] Table 4: Configuration of noise estimation components. 'C' denotes a convolutional layer and 'TC' denotes a replaced convolutional layer.
[0088] The denoising component of our model consists of two 2D convolutional encoders, a bidirectional LSTM, and three FC layers. The two convolutional encoders take as input the input speech spectrogram S x and the estimated full noise spectrogram
number
[0089] [Table 5] [Table 5] Table 5: Convolutional encoders for the denoising component of our model. Each convolutional layer is followed by a batch normalization layer with the ReLU activation function as the activation function.
[0090] Figure 7: Noisy audio constructed based on different SNR levels. The first column shows the waveform of the ground truth clean input.
[0091] Training details. We use the PyTorch platform to implement our speech denoising model, which is then trained with the Adam optimizer. In our end-to-end training without silent interval supervision (referred to as "our w / o SID loss" in Sect. 4 and see Sect. 3.2), we run the Adam optimizer with a batch size of 20 and a learning rate of 0.001 for 50 epochs. When silent interval supervision is incorporated (recall Sect. 3.3), we first train the silent interval detection component with the following settings: run the Adam optimizer with a batch size of 15 and a learning rate of 0.001 for 100 epochs. Later, we train the noise estimation and removal components using the same settings as the end-to-end training of "our w / o SID loss".
[0092] (B: Details of data processing) Our model is designed to take mono-channel audio clips of any length as input. However, when constructing the training dataset, we set the audio clips in the training dataset to have the same length of 2 seconds each, allowing for batches at training time. For this purpose, we split each original audio clip from AVSPEECH, DEMAND, and AudioSet into clips of 2 seconds length. The whole audio clip is then downsampled at 16 kHz and converted to a spectrogram using STFT. To perform STFT, the size of the Fast Fourier Transform (FFT) is set to 510, the size of the Hann window is set to 28 ms, and the hop length is set to 11 ms. As a result, each 2-second clip has a spectrogram with a resolution of 256x178 (complex values), where 256 is the number of frequency bins and 178 is the time resolution. At the estimation time, our model can still accept audio clips of any length.
[0093] Both our clean speech dataset and noisy dataset are first split into training and testing sets, so that none of the training and testing audio clips come from the same original audio source - they are well separated.
[0094] To monitor our silent interval detection, we classify the clean audio signal in the following way: We first normalize each audio clip so that its magnitude is in the range [-1,1], i.e., ensuring the maximum waveform magnitude at -1 or 1. Then, the clean audio clip is divided into segments of 1 / 30 seconds in length. We classify a time section as a "silent" segment (i.e., label 0) if the average waveform energy is below 0.08 in that segment; otherwise, it is classified as a "non-silent" segment (i.e., label 1).
[0095] (C: Evaluation of silent interval detection): C.1: Metrics We now provide details of the metrics utilized to evaluate our silence interval detection (i.e., results in Table 1). Detecting silence intervals is a binary classification task: classify the entire time segment as either silent (i.e., the positive condition) or not (i.e., the negative condition). Recall that the confusion matrix for a binary classification task looks like this:
[0096] [Table 6] [Table 6] Table 6: Confusion matrix
[0097] In our case, we have the following conditions: True positive (TP) samples are silence segments that are correctly predicted. True negative (TN) samples are non-silence segments that are correctly predicted. False positive (FP) samples are non-silence segments that are predicted as silence. False negative (FN) samples are silence segments that are predicted as non-silence. The four metrics utilized in Table 1 follow standard definitions of statistics, which we review here:
[0098] [Number 1]
number
[0099] In the formula, N TP , N TN , N FP , and N FN denotes the number of true positive, true negative, false positive, and false negative predictions among all tests. Intuitively, recall indicates the ability to correctly find all true silence intervals, and precision measures what proportion of classified silence intervals are truly silent. The F1 score takes precision and recall into account and produces their harmonic average, and precision is the proportion of correct predictions among all predictions.
[0100] (C.2: Example of silent interval detection) In Figure 9, we present an example of silent interval detection results in a comparison of two alternative methods. The two alternatives are described in Section 4.3 and are referred to as Baseline Threshold and VAD, respectively. Figure 9 repeats the quantitative results of Table 1: VAD tends to be overly conservative even in the presence of light noise; many silent intervals are ignored. On the other hand, Baseline Threshold tends to be overly aggressive; it generates many spurious intervals. In contrast, our silent interval detection maintains a better balance and therefore predicts more accurately.
[0101] Figure 9: An example of the result of silent interval detection. Given an input signal with an SNR of 0 dB (top left), we show silence intervals (red) detected by three approaches: our method, the baseline threshold, and the VAD. We also show the ground truth silence intervals (top left).
[0102] (D: Ablation study and analysis) (D.1: Details of ablation study) In Section 4.4 and Table 2, an ablation study is set up in the following way: "Ours" refers to our proposed network structure and training method that incorporates silent interval monitoring (recall Section 3.3). Details are given in A. "Our w / o SID loss" refers to our proposed network structure, but optimized by the training method of Section 3.2 (i.e., end-to-end training without silent interval monitoring). This ablation study confirms that silent interval monitoring is indeed helpful in improving the quality of denoising. "Our joint loss" refers to the proposed network structure optimized by an end-to-end training approach that optimizes the loss function (1) with an additional term (2). In this end-to-end training, silence interval detection is also monitored by the loss function. This ablation study confirms that our two-stage training (Section 3.3) is more effective. "Our w / o NE loss" utilizes our two-stage training (Sect. 3.3) but without the noise estimation loss term - i.e., the first term in (1) is missing. This ablation study is to examine the necessity of the noise estimation loss term for better denoising quality. "Our w / o SID comp" turns off the silence interval detection: the silence interval detection component always outputs an all-zero vector. As a result, the input noise profile for the noise estimation component N is made exactly the same as the original noisy signal. This ablation study is to examine the effect of silence intervals for speech denoising. "Our w / o NR comp" utilizes simple spectral subtraction to replace our denoising component. The other components remain "unaltered". This ablation study is to examine the effectiveness of our denoising component.
[0103] (D.2: Impact of silent interval detection on noise reduction quality) A key insight of our neural network-based denoising model is the exploitation of the distribution of silence intervals over time. The experiments above confirm the effectiveness of our silence interval detection for better speech denoising. We now report additional experiments, aiming to gain some empirical understanding of how the quality of silence interval prediction affects the quality of speech denoising.
[0104] First, starting with ground truth silence intervals, we shift them by 1 / 30, 1 / 10, 1 / 6, and 1 / 2 seconds. As the shift length increases, more time segments are incorrectly classified: the number of both false positive labels (i.e., non-silent time segments classified as silence) and false negative labels (i.e., silent time segments classified as non-silent) increases. After each shift, we feed the silence interval labels into our noise estimation and removal component and measure the quality of the noise removal under a PESQ score.
[0105] In the second experiment, we again start with ground truth silence intervals; however, instead of varying them, we shrink the silence intervals toward their centers by 20%, 40%, 60%, and 80%, respectively. The more the silence intervals shrink, the fewer time segments are classified as silent. In other words, the number of false negative predictions only increases. As in the previous experiment, after each shrink, we use the silence interval labels in our speech denoising pipeline and measure the PESQ score.
[0106] The results of both experiments are reported in Table S5. When we reduced the silence interval, the quality of the denoising slightly decreased. In contrast, even small changes caused a clear decrease in the quality of the denoising. These results suggest that false positive predictions have a more negative impact on the quality of the denoising compared to false negative predictions. On the other hand, a reasonably conservative prediction may leave certain silent time segments undetected (i.e., the incorporation of some false negative labels), while the detected silent intervals actually reveal noise profiles. On the other hand, even a small amount of false positive predictions may lead certain non-silent time segments to be treated as silent segments, so that the observed noise profile in the detected silent intervals is overshadowed by the foreground signal.
Claims
1. A computer-implemented method comprising: receiving an audio signal representation; Detecting, using a first learned model, one or more silence intervals in the received audio signal representation, in which the foreground sound level is reduced; determining an estimated full noise profile corresponding to the audio signal representation based on the detected one or more silence intervals; and generating a resultant audio signal representation having a reduced noise level based on the received audio signal representation and the determined estimated full noise profile using a second learning model. A method for providing the above.
2. Detecting the one or more silence intervals utilizing the first learned model includes: dividing the audio signal representation into a plurality of segments, each segment being shorter than the length of an interval of the received audio signal representation; converting the plurality of segments into a time-frequency representation; and 2. The method of claim 1, comprising: utilizing a first learning machine to process the time-frequency representations of the plurality of segments to implement the first learning model to generate, for each of the plurality of segments, a noise vector including a confidence value representing the likelihood that each one of the plurality of segments is a silent interval.
3. The step of processing the time-frequency representation comprises: encoding the time-frequency representations of the plurality of segments with a 2D convolutional encoder to generate a 2D feature map; applying a learning network structure, including at least a bidirectional long short-term memory (LSTM) structure, to the 2D feature map to generate a silence vector; determining a noise mask from the silence vector; and The method of claim 2 , comprising generating a partial noise profile of the audio signal representation based on the audio signal representation and the noise mask.
4. The step of determining the estimated full noise profile comprises: generating a partial noise profile representative of time-frequency characteristics of the detected silence interval or intervals; converting the audio signal representation and the partial noise profile into respective time-frequency representations; applying a convolutional encoding to the audio signal representation and the time-frequency representation of the partial noise profile to generate an encoded audio signal representation and an encoded partial noise profile; and The method of claim 1 , further comprising combining the encoded audio signal representation and the encoded partial noise profiles to generate the estimated full noise profile.
5. The step of generating the resulting audio signal representation having the reduced noise level comprises: generating a time-frequency representation of the audio signal representation and the estimated full noise profile; and 5. The method of claim 1 , comprising applying the second learning model to the audio signal representation and the time-frequency representation of the estimated full noise profile to generate the resulting audio signal representation.
6. The method of claim 5 , wherein the second learning model is implemented with a bidirectional long short-term memory (LSTM) structure.
7. A program for causing a computer to execute the method according to any one of claims 1 to 6.
8. A receiver unit for receiving an audio signal representation; and Implementing one or more learning engines in communication with the receiver unit and a memory device for storing programmable instructions, Detecting one or more silence intervals in the received audio signal representation utilizing a first learned model, the silence intervals having a reduced foreground sound level; determining an estimated full noise profile corresponding to the audio signal representation based on the detected one or more silence intervals; and generating a resultant audio signal representation having a reduced noise level based on the received audio signal representation and the determined estimated full noise profile using a second learning model. Controller that does A system comprising:
9. receiving an audio signal representation; Detecting one or more silence intervals having a reduced foreground sound level in the received audio signal representation utilizing a first learned model; determining an estimated full noise profile corresponding to the audio signal representation based on the detected one or more silence intervals; and generating a resultant audio signal representation having a reduced noise level based on the received audio signal representation and the determined estimated full noise profile using a second learning model. A non-transitory computer-readable medium storing a set of instructions executable on at least one programmable device to perform the steps of:
Citation Information
Patent Citations
Voice pass filter
JP1990253298A
Voice coding method
JP1994282297A
