Neural noise reduction with linear and non-linear filtering for single-channel audio signals

Abstract

The invention provides a method, a device and a system for audio signal processing. The present implementation more specifically relates to speech enhancement techniques that combine statistical signal processing with neural network inference. In some aspects, a speech enhancement system may include a linear filter, a deep neural network (DNN), and a non-linear post filter. The linear filter and the non-linear post filter are configured to suppress noise in an audio signal using statistical signal processing techniques. More specifically, the linear filter de-noising an input audio signal based on a temporal correlation between consecutive frames of the audio signal. The DNN inferes a speech signal and a noise signal (respectively representing speech and noise components of the audio signal) based on the de-noised audio signal. The non-linear post filter suppresses residual noise in the speech signal based on one or more Gaussian mixture models (GMMs) associated with the speech signal and the noise signal.

Description

[0001] Cross-reference to related applications This patent application claims priority to U.S. Non-Provisional Patent Application No. 18 / 310818, filed May 2, 2023, entitled “NEURAL NOISE REDUCTION WITH LINEAR AND NONLINEARFILTERING FOR SINGLE-CHANNEL AUDIO SIGNALS”, which has been assigned to its assignee and is incorporated herein by reference in its entirety. Technical Field

[0002] This implementation generally involves signal processing, and specifically involves neural noise reduction techniques using linear and nonlinear filtering for single-channel audio signals. Background Technology

[0003] Many hands-free communication devices include a microphone configured to convert sound waves into an audio signal that can be transmitted to a receiving device via a communication channel. The audio signal typically includes a speech component (such as from the user of the communication device) and a noise component (such as from a reverberation enclosure). Speech enhancement is a signal processing technique that attempts to suppress the noise component of a received audio signal without distorting the speech component. Many existing speech enhancement techniques rely on statistical signal processing algorithms that continuously track noise patterns in each frame of the audio signal to model a spectral suppression gain or filter that can be applied to the received audio signal in the time-frequency domain.

[0004] Some modern speech enhancement techniques incorporate machine learning to model spectral suppression gains or filters that can be applied to received audio signals in the time-frequency domain. Machine learning, generally comprising a training phase and an inference phase, is a technique used to improve the ability of a computer system or application to perform a task. During the training phase, the machine learning system is provided with one or more "answers" and a large amount of raw training data associated with those answers. The machine learning system analyzes the training data to learn a set of rules that can be used to describe each of the one or more answers. During the inference phase, the machine learning system uses the learned set of rules to infer answers from new data.

[0005] Deep learning is a specific form of machine learning where the inference (and training) phases are performed across multiple layers, resulting in a more abstract dataset at each successive layer. Because information is processed in a way that is analogous to a biological nervous system, deep learning architectures are often referred to as "artificial neural networks." For example, each layer of an artificial neural network can consist of one or more "neurons." Neurons can be interconnected across various layers, allowing input data to be processed and passed from one layer to another. More specifically, each layer of neurons can perform different transformations on the output data from the previous layer, such that the final output of the neural network leads to the desired inference. The collection of transformations associated with the various layers of the network is called a "neural network model."

[0006] The size of a neural network (such as the number of layers or neurons in each layer) generally affects the accuracy of inference results. More specifically, larger neural networks tend to produce more accurate inferences compared to smaller or more compact ones. However, speech enhancement for single-channel audio is typically implemented using low-power edge devices with very limited resources, such as battery-powered headphones, earbuds, and other hands-free communication devices with a single microphone input. Consequently, many existing single-channel speech enhancement techniques rely on compact neural network architectures that produce filtered audio signals with a certain amount of speech distortion or noise leakage (also known as "residual noise"). Therefore, there is a need to improve the quality of speech in single-channel audio signals. Summary of the Invention

[0007] This summary is provided to introduce, in a simplified form, the selection of concepts further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0008] One innovative aspect of the subject matter of this disclosure can be implemented in a speech enhancement method. The method includes the steps of: receiving a series of frames of an audio signal; denoising a first frame in the series of frames based at least in part on the temporal correlation between the frames; inferring a speech probability associated with the denoised first frame based on a neural network model; generating a first speech signal and a first noise signal based on the speech probability associated with the denoised first frame, wherein the first speech signal and the first noise signal represent the speech component and noise component of the audio signal in the first frame, respectively; determining a first spectral suppression gain based on the first speech signal and the first noise signal; and suppressing residual noise in the first speech signal based on the first spectral suppression gain.

[0009] Another innovative aspect of the subject matter of this disclosure can be implemented in a speech enhancement system comprising a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the speech enhancement system to: receive a series of frames of an audio signal; denoise a first frame in the series of frames based at least in part on the temporal correlation between the frames; infer a speech probability associated with the denoised first frame based on a neural network model; generate a first speech signal and a first noise signal based on the speech probability associated with the denoised first frame, wherein the first speech signal and the first noise signal represent the speech component and noise component of the audio signal in the first frame, respectively; determine a first spectral suppression gain based on the first speech signal and the first noise signal; and suppress residual noise in the first speech signal based on the first spectral suppression gain. Attached Figure Description

[0010] This implementation is illustrated by way of example and is not intended to be limited by the figures in the accompanying drawings.

[0011] Figure 1 An example audio receiver that supports single-channel voice enhancement is shown.

[0012] Figure 2 A block diagram of an example speech enhancement system based on some implementation methods is shown.

[0013] Figure 3 A block diagram of an example nonlinear filter for a single-channel audio signal is shown, according to some implementations.

[0014] Figure 4 Another block diagram of an example speech enhancement system based on some implementations is shown.

[0015] Figure 5 An illustrative flowchart is shown, depicting example operations for processing audio signals according to some implementations. Detailed Implementation

[0016] In the following description, numerous specific details (such as examples of specific components, circuits, and processes) are set forth to provide a thorough understanding of this disclosure. As used herein, the term “coupled” means a direct connection to or through one or more intermediate components or circuits. The terms “electronic system” and “electronic device” can be used interchangeably to refer to any system capable of electronically processing information. Furthermore, in the following description, and for purposes of explanation, specific terminology is set forth to provide a thorough understanding of aspects of this disclosure. However, it will be apparent to those skilled in the art that these specific details are not required to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring this disclosure. Some portions of the subsequent detailed description are presented in accordance with other notations of procedures, logic blocks, processes, and operations on data bits within computer memory.

[0017] These descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to others skilled in the art. In this disclosure, procedures, logic blocks, processes, etc., are considered as a self-consistent sequence of steps or instructions that lead to a desired result. These steps are those that require physical manipulation of physical quantities. Typically, although not always, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system. However, it should be remembered that all terms in these and similar terms are to be associated with appropriate physical quantities and are merely convenient notations applied to those quantities.

[0018] Unless otherwise specifically stated, as will be apparent from the following discussion, throughout this application, the use of terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” etc., relates to the operation and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the registers and memories of the computer system into other data similarly represented as physical quantities in the computer system’s memory or registers or other such information storage, transmission, or display devices.

[0019] In the accompanying drawings, a single block may be described as performing one or more functions; however, in practice, the one or more functions performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described below according to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure. Furthermore, example input devices may include components different from those shown, including known components such as processors, memory, etc.

[0020] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any feature described as a module or component may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging material.

[0021] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Alternatively or additionally, the technology may be implemented at least in part by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures and is accessible, read, and / or executed by a computer or other processor.

[0022] The various illustrative logic blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein can be executed by one or more processors (or processing systems). As used herein, the term "processor" can mean any general-purpose processor, special-purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0023] As described above, some modern speech enhancement techniques utilize neural networks to model spectral suppression gains or filters that can be applied to audio signals in the time-frequency domain. Generally, larger neural networks tend to produce more accurate inferences compared to smaller or more compact ones. However, speech enhancement for single-channel audio is typically implemented using low-power edge devices with limited resources, such as battery-powered headphones, earbuds, and other hands-free communication devices with a single microphone input. Accordingly, many existing single-channel speech enhancement techniques rely on compact neural network architectures that produce filtered audio signals with a certain amount of speech distortion or noise leakage (also known as “residual noise”). This disclosure recognizes that statistical signal processing techniques can be combined with neural network inference to further improve the quality of speech in single-channel audio signals.

[0024] Various aspects generally involve audio signal processing, and more specifically, speech enhancement techniques that combine statistical signal processing with neural network inference. In some aspects, speech enhancement systems may include linear filters, deep neural networks (DNNs), and nonlinear post-filters. Linear filters and nonlinear post-filters are configured to use statistical signal processing techniques to suppress noise in the audio signal. More specifically, the linear filter denoises the input audio signal based on the temporal correlation between consecutive frames of the audio signal. The DNN infers the speech probability in the denoised audio signal and generates a speech signal and a noise signal (representing the speech component and noise component of the audio signal, respectively) based on the inferred speech probability. In some implementations, the speech probability can also be used to update various parameters of the linear filter (such as the weight vector associated with a multi-frame beamformer). The nonlinear post-filter suppresses residual noise in the speech signal based on a Gaussian mixture model (GMM) associated with the speech and noise signals.

[0025] Specific implementations of the subject matter described in this disclosure can achieve one or more of the following potential advantages. By combining statistical signal processing with neural network inference, aspects of this disclosure can significantly improve the speech quality of hands-free communication devices. More specifically, a linear filter pre-adjusts the audio signal before it is input to the DNN to reduce noise, while a nonlinear post-filter further suppresses residual noise in the audio signal output by the DNN. Accordingly, the speech enhancement system of this implementation can achieve similar inference results to much larger neural networks using a relatively compact neural network. Because statistical signal processing techniques require relatively low overhead (compared to larger neural networks), the speech enhancement system of this implementation is well-suited for implementation in low-power edge devices with very limited resources.

[0026] Figure 1An example audio receiver 100 supporting single-channel speech enhancement is shown. The audio receiver 100 includes a microphone 110 and a speech enhancement component 120. The microphone 110 is configured to convert sound waves 101 (also referred to as “acoustic waves”) into an audio signal 102. Thus, the audio signal 102 is an electrical signal representing an acoustic waveform. In some aspects, the microphone 110 may be associated with a single audio channel. Therefore, the audio signal 102 may also be referred to as a “single-channel” audio signal.

[0027] In some implementations, sound wave 101 may include user speech mixed with background noise or interference, such as reverberation noise from the headset housing. Therefore, audio signal 102 may include speech components and noise components. For example, audio signal 102 ( ) can be expressed as speech components ( ) and noise components ( The combination of ) in which It is a frame index, and, It is a frequency index associated with the time-frequency domain: The voice enhancement component 120 is configured, for example, to suppress noise components. Alternatively, the signal-to-noise ratio (SNR) of the audio signal 102 can be increased in other ways to improve the quality of speech in the audio signal 102. In some implementations, the speech enhancement component 120 may apply a spectrum suppression gain or filter to the audio signal 102. The spectrum suppression gain reduces the noise components of the audio signal 102 in the time-frequency domain. The power is attenuated to produce an enhanced speech signal 104. As a result, the enhanced speech signal 104 can have a higher SNR than the audio signal 102.

[0028] In some aspects, the speech enhancement component 120 may determine the spectral suppression gain based at least in part on a deep neural network (DNN) 122. For example, the DNN 122 may be trained to infer the likelihood or probability of speech in an audio signal. Among other examples, suitable DNNs include convolutional neural networks (CNNs) and recurrent neural networks (RNNs). During the training phase, the DNN 122 may be provided with a large audio signal containing speech mixed with background noise (also referred to as "noisy speech" signal). The DNN 122 may also be provided with a clean speech signal (without background noise) representing only the speech component of each audio signal. The DNN 122 compares the noisy speech signal with the clean speech signal to determine a set of features that can be used to classify the speech.

[0029] During the inference phase, DNN 122 can index each frequency in the time-frequency domain based on the classification results. Next, determine each frame of the audio signal 102. The probability of speech in ( In some implementations, DNN 122 can further incorporate speech probabilities. Converted into a signal that can be used to generate enhanced speech signals. ) spectral suppression gain ( ),in More specifically, spectral suppression gain Can suppress the first Noise components of audio signal 102 in each audio frame For example, if in the first Under the frequency index, in the audio signal 102, the... If a frame has a low speech probability (indicating that noise is dominant at that time-frequency index), then... The value may be relatively low, making it difficult to apply the spectral suppression gain to the audio signal 102. The power is attenuated.

[0030] As described above, the size of a neural network (such as the number of layers or neurons in each layer) generally affects the accuracy of inference results. More specifically, larger neural networks tend to produce more accurate inferences compared to smaller or more compact ones. Consequently, existing neural network architectures require significant processing power and memory to achieve accurate speech enhancement (especially for single-channel audio signals). However, single-channel speech enhancement is typically used in low-power edge devices with limited resources, such as battery-powered headsets, earbuds, and other hands-free communication devices with a single microphone input. Therefore, for many single-channel speech enhancement applications, compact neural networks may be more suitable than larger ones.

[0031] In some implementations, DNN 122 can be a relatively compact neural network. As a result, DNN 122 may not filter at least some of the noise in the audio signal 102. In other words, DNN 122 can generate an enhanced speech signal with some speech distortion or residual noise. This disclosure recognizes that statistical signal processing techniques can be combined with neural network inference to further improve the quality of speech in a single-channel audio signal. Among other examples, suitable statistical processing techniques include linear and nonlinear filtering techniques. In some aspects, the speech enhancement component 120 may perform linear filtering on the input of the DNN 122 to suppress noise in the audio signal 102 before it is processed by the DNN 122. In some other aspects, the speech enhancement component 120 may perform nonlinear filtering on the output of the DNN 122 to further suppress residual noise in the enhanced speech signal 104.

[0032] Figure 2 A block diagram of an example speech enhancement system 200 according to some implementations is shown. In some implementations, the speech enhancement system 200 may be... Figure 1 An example of a speech enhancement component 120. More specifically, the speech enhancement system 200 can receive a series of frames of an input audio signal ( And by filtering or suppressing noise in the audio signal, corresponding frames of the enhanced audio signal are generated. (For example, see reference) Figure 1 The input audio signal can be an example of a single-channel audio signal 102, and the enhanced audio signal can be an example of an enhanced speech signal 104.

[0033] A series of input audio frames Including the current audio frame to be processed (( In time, at the current audio frame A certain number (c) of future audio frames ( ) and in time at the current audio frame A certain number (d) of previous audio frames ( ), so that: in Furthermore, Δ is the value that determines a series of input audio frames. The delay parameter Δ is the delay between consecutive frames in the audio input. In some implementations, the delay parameter Δ can be set to a value smaller than the frame hop (such as 1 sample) associated with the speech enhancement system 200 to ensure sufficient latency across input audio frames. Temporal speech correlation.

[0034] For example, given The size of the Fast Fourier Transform (FFT) (where It is the number of frequency bins associated with the FFT, and the current audio frame. It can be expressed as: Furthermore, previous audio frames It can be expressed as: Furthermore, future audio frames It can be expressed as: In some implementations, the speech enhancement system 220 may include a linear filter 210, a DNN 220, and a nonlinear post-filter 230. The linear filter 210 is configured to be based on a series of input audio frames. To generate denoised audio frames More specifically, the linear filter 210 may be based on a series of input audio frames. Associated temporal correlation to suppress the current audio frame noise components ( (or attenuate it.) In some implementations, the linear filter 210 may include a multi-frame beamformer. Examples of suitable multi-frame beamformers include, but are not limited to, multi-frame minimum variance distortionless response (MF-MVDR) beamformers.

[0035] Multi-frame beamforming utilizes the temporal characteristics of a single-channel audio signal to enhance speech. More specifically, multi-frame beamforming relies on the accurate prediction or estimation of the temporal correlation of speech between consecutive audio frames (also known as "inter-frame correlation of speech"). For example, referring to Equation 1, the speech components of the audio signal... It can be decomposed into related parts ( ) and unrelated parts ( ): in It is with audio frames The inter-frame correlation (IFC) vector associated with the speech components. is a matrix representing the covariance of the speech components, and It is a choice The vector of the first column. Accordingly, the multi-frame signal model can be expressed as: Among the unrelated speech components It is considered a disturbance.

[0036] For example, multi-frame beamformers can use IFC vectors To make a series of input frames Alignment, so that in the input frame When summed together, the speech components Combine (or noise components) in a constructive manner (Combined in a destructive manner). For example, an MF-MVDR beamformer can combine weight vectors... Applied to a series of audio frames To generate denoised audio frames : In some respects, the linear filter 210 can determine the denoised audio frames. Weight vector relative to one or more conditions for optimization For example, linear filter 210 can determine the audio frame without making it... Reduce audio frames in the case of speech component distortion The variance of the noise components or the weight vector that minimizes that variance. In other words, the weight vector The following conditions must be met: in It represents an audio frame. The matrix of covariance of the noise components. The resulting weight vector. This indicates an MF-MVDR beamforming filter ( It can be expressed as: In some implementations, the linear filter 210 can convert the IFC vector and noise covariance matrix As The function is estimated or tracked over time. More specifically, the linear filter 210 can update the IFC vector when speech is present in the input audio signal or otherwise detected. Furthermore, it can avoid updating the IFC vector when speech is absent from the input audio signal or is otherwise undetected. On the other hand, the linear filter 210 can update the noise covariance matrix when speech is absent from the input audio signal or is otherwise undetected. Furthermore, it avoids updating the noise covariance matrix when speech is present in or detected in the input audio signal. .

[0037] The DNN 220 is configured to infer the current audio frame based on a neural network model. Speech probability in , where 0≤ ≤1. In some implementations, DNN 220 can be... Figure 1An example of DNN 122. In some respects, linear filter 210 may be based at least in part on speech probabilities inferred by DNN 220. To update the IFC vector or noise covariance matrix For example, the linear filter 210 can be based on speech probability. To determine whether speech is present or absent in the audio signal (also known as speech activity detection (VAD)) and therefore whether the IFC vector should be updated. or noise covariance matrix Accordingly, the linear filter 210 can be based on previous audio frames. The speech probability is used to determine which audio frame to apply to. weight vector .

[0038] In some aspects, DNN 220 can be further based on speech probability. To generate speech signals ( ) and noise signals ( (including speech signals) Represents denoised audio frames The speech components, and the noise signal Represents denoised audio frames The noise component. For example, DNN 220 can be based on speech probability. To calculate the spectral suppression gain ( ), and can reduce spectral suppression gain Applied to denoised audio frames To generate speech signals ,in Noise signal Can be calculated as denoised audio frames With voice signal The difference between them, of which .

[0039] In some respects, the DNN 220 may be biased towards minimizing speech distortion rather than maximizing noise suppression. In other words, the spectral suppression gain... It can be tuned to ensure denoised audio frames The speech components in the obtained speech signal Without distortion. For example, DNN 220 can transmit speech signals without loss of quality. As a denoised audio frame Voice probability And the tuning parameters that control the amount of noise reduction via DNN 220 ( The operation is performed on a function of ). in It is a voice signal amplitude, It is a voice signal The phase, and 0≤ <1. In some implementations, tuning parameters Configurable to make the voice signal It does not contain speech distortion (and therefore may contain some residual noise).

[0040] exist Figure 2 In the example, the neural network model was trained to infer speech probabilities In some other implementations, neural network models can be trained to infer denoised audio frames. speech components In this type of implementation, the probability of speech... and voice signals Can be used as a denoised audio frame and the inferred speech components Calculated using a function: The nonlinear post-filter 230 is configured to be based on the speech signal. and noise signals To generate enhanced audio frames More specifically, the nonlinear post-filter 230 can be based on the speech signal. and noise signals One or more associated speech activity detection (VAD) features to make the speech signal Residual noise attenuation in the audio frame. As used herein, the term "VAD feature" refers to the feature that reflects the current audio frame. The presence of speech signals in speech noise signal Any characteristic of or any combination thereof. In some implementations, one or more VAD features may include speech signals. With noise signal Normalized difference between them ).

[0041] This disclosure recognizes that voice signals It mainly contains the target speech, and also noise signals. It mainly includes background noise. Therefore, the signal... and Normalized difference between The target speech may exist in the speech signal. The value is closer to +1 when the target speech is not present in the speech signal. It is closer to -1: In some implementations, the nonlinear post-filter 320 can use an online Gaussian mixture model (GMM) based on the speech signal. With noise signal Normalized difference between To determine the relationship with the speech signal Associated speech probability ( For example, normalized difference. It can be used to create bimodal models with two Gaussian probability density functions (PDFs), namely a Gaussian PDF dominated by the target speech and a Gaussian PDF dominated by noise. The online GMM can be used to compute weights for each Gaussian PDF. ), mean ( ) and variance ( ),in =1 indicates that the target speech-dominant Gaussian PDF is used, and, =2 indicates a noise-dominant Gaussian PDF: in It represents the soft speech probability under each time-frequency index. ,and, It is the learning rate step size.

[0042] In some implementations, step size Capable of inferring denoised audio frames from DNN 220 and voice probability Determined from adaptive location: in It is the maximum step size that can be used for subband parameter tracking (and can be a tunable hyperparameter associated with the speech enhancement system 200), and, to This indicates the frequency range where speech is dominant (e.g., 0-2kHz).

[0043] In some implementations, VAD (Video Array Controller) is based on multi-frame time processing. It can be expressed as the current audio frame. and denoised audio frames Functions: in, , , as well as It is the tuning threshold (between 0 and 1). In some implementations, It can be used to indicate which sets of GMM parameters should be updated (such as for speech or noise): In some implementations, the nonlinear post-filter 230 may be based on speech probability. To estimate speech signals The amplitude (or power) of the noise in ( ), and use spectral subtraction to determine which can be applied to speech signals. To produce enhanced audio frames Spectral suppression gain ( ): in It is a tuning parameter representing the floor gain associated with spectral subtraction.

[0044] This disclosure recognizes that multiple VAD features (including speech signals) can be enabled. With noise signal Normalized difference between Combining these elements to produce a more robust spectral suppression gain for different noise environments. Other suitable VAD features may include speech signals. The cepstral peak, spectral entropy, and harmonic product spectrum (HPS), among other examples. In some implementations, the nonlinear post-filter 230 can determine the characteristic of each VAD (VaD). The corresponding speech probability associated with ) )(in ≥1), and based on VAD features Speech probabilities associated with any VAD feature The lowest probability is used to calculate the spectral suppression gain. .

[0045] Figure 3 A block diagram of an example nonlinear filter 300 for a single-channel audio signal is shown, according to some implementations. In some implementations, the nonlinear filter 300 may be... Figure 2 An example of a nonlinear filter 230. More specifically, the nonlinear filter 300 may be based on a speech signal. and noise signals To generate enhanced audio frames For reference Figure 2 The described speech signal and noise signals They can represent audio frames respectively. The speech components and noise components, among which In some implementations, signals and It can be generated by neural networks (such as Figure 2 (The DNN 220) is inferred or generated in other ways.

[0046] In some respects, the nonlinear filter 300 can be based on the speech signal. and noise signals Associated VAD features of a certain number (M) to suppress speech signals Residual noise in the signal. In some implementations, the nonlinear filter 300 can suppress the speech signal based on multiple VAD features (M>1). Residual noise in. Figure 3 In the example, the nonlinear filter 300 is shown as comprising M feature extractors 310(1)–310(M), M GMMs 320(1)–320(M), and a noise suppressor 330. Each feature extractor in the feature extractors 310(1)–310(M) is configured to be based on the speech signal. noise signal or any combination thereof to calculate the corresponding VAD features ,in .

[0047] In some implementations, at least one of the feature extractors 310(1)–310(M) can compute the speech signal. With noise signal The normalized difference between them. For example, the first VAD feature ( It can be calculated according to Equation 3. Although each feature extractor in feature extractors 310(1)–310(M) is shown as receiving speech signals and noise signals However, some feature extractors can be based solely on speech signals. To calculate the corresponding VAD features For example, VAD features One or more VAD features in the speech signal can be based on the speech signal. Calculated using the power spectrum: in It is a smoothing factor (between 0 and 1), and, It is the noise power spectrum, which can be obtained through pauses in speech (such as when speech is in the speech signal). When not detected in the power spectrum Estimate by taking the average.

[0048] In some implementations, at least one of the feature extractors 310(1)–310(M) can compute the feature with respect to the speech signal. Associated cepstral peaks. For example, the second VAD feature ( ) can be calculated as: in Indicates the relationship with the first The first frame The lag associated with each sample, and, It is related to speech signals The total number of associated frequency windows.

[0049] In some implementations, at least one of the feature extractors 310(1)–310(M) can compute the feature with respect to the speech signal. The associated spectral entropy. For example, the third VAD feature ( ) can be calculated as: in It is related to speech signals The total number of associated frequency windows.

[0050] In some implementations, at least one of the feature extractors 310(1)–310(M) can compute the feature with respect to the speech signal. Associated harmonic product spectrum (HPS). For example, the fourth VAD feature ( ) can be calculated as: in Is with The corresponding number of harmonic components, Indicates the relationship with the first The first frame The lag associated with each sample, and, It is related to speech signals The total number of associated frequency windows.

[0051] Each GMM in GMM 320(1)–320(M) is configured based on the corresponding VAD feature. To determine the relationship with the speech signal Associated speech probability For example, VAD features Each VAD feature in the model can be used to create a corresponding bimodal model with two Gaussian PDFs, namely a Gaussian PDF dominated by the target speech and a Gaussian PDF dominated by noise. In some implementations, each GMM in GMM 320(1)–320(M) can be based on the corresponding VAD feature. Weights of each associated Gaussian PDF mean and variance To calculate the probability of the corresponding speech (such as references) Figure 2 (And described). For example, the probability of each speech. It can be calculated based on Equation 4.

[0052] The noise suppressor 330 is configured to be at least partially based on VAD characteristics, respectively. Associated speech probability To generate enhanced audio frames In some implementations, the noise suppressor 330 can suppress speech probability. Select the lowest speech probability ( ),in And based on the lowest speech probability To calculate the spectral suppression gain For example, the noise suppressor 330 can use speech probability according to Equation 5. To process speech signals The amplitude (or power) of the noise in the image. Furthermore, it can be determined based on the noise amplitude (or power) according to Equation 6. To calculate the spectrum suppression gain The noise suppressor 330 can further increase the spectral suppression gain. Applied to speech signals To produce enhanced audio frames ,in .

[0053] Figure 4 Another block diagram of an example speech enhancement system 400 according to some implementations is shown. More specifically, the speech enhancement system 400 may be configured to receive a single-channel audio signal and generate an enhanced audio signal by filtering or suppressing noise in the received audio signal. In some implementations, the speech enhancement system 400 may be... Figure 1 An example of a voice enhancement component 120. The voice enhancement system 400 includes a device interface 410, a processing system 420, and a memory 430.

[0054] Device interface 410 is configured to interface with one or more components of an audio receiver (such as...) Figure 1 Communication is via microphone 110. In some implementations, device interface 410 may include a microphone interface (I / F) 412 configured to receive a single-channel audio signal via the microphone. In some implementations, microphone interface 412 may sample or receive individual frames of the audio signal at frame jumps associated with the speech enhancement system 400. For example, a frame jump may represent the frequency at which an application requires or otherwise expects to receive enhanced audio frames from the speech enhancement system 400.

[0055] Memory 430 may include audio data storage 432, configured to store a series of frames of audio signals and any intermediate signals generated by the speech enhancement system 400 as a result of performing speech enhancement operations. Memory 430 may also include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard disk drives, and other examples) capable of storing at least the following software (SW) modules: • Linear filtering SW module 434, which is used to denoise the first frame in a series of frames based at least in part on the temporal correlation between the series of frames; • DNN SW module 436, which is used to infer the speech probability associated with the denoised first frame based on a neural network model and to generate a speech signal and a noise signal based on the speech probability associated with the denoised first frame, wherein the speech signal and the noise signal represent the speech component and noise component of the audio signal in the first frame, respectively; and • Nonlinear filtering SW module 438, which is used to determine the spectrum suppression gain based on the speech signal and the noise signal, and to suppress residual noise in the speech signal based on the spectrum suppression gain.

[0056] Each software module includes instructions that, when executed by the processing system 420, cause the speech enhancement system 400 to perform a corresponding function.

[0057] Processing system 420 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in speech enhancement system 400 (such as in memory 430). For example, processing system 420 may execute a linear filtering SW module 434 to denoise the first frame in a series of frames, at least in part, based on the temporal correlation between the series of frames. Processing system 420 may also execute a DNN SW module 436 to infer the speech probability associated with the denoised first frame based on a neural network model and to generate a speech signal and a noise signal based on the speech probability associated with the denoised first frame, wherein the speech signal and the noise signal represent the speech component and noise component of the audio signal in the first frame, respectively. Further, processing system 420 may execute a nonlinear filtering SW module 438 to determine a spectral suppression gain based on the speech signal and the noise signal and to suppress residual noise in the speech signal based on the spectral suppression gain.

[0058] Figure 5 An illustrative flowchart depicting example operation 500 for processing audio signals according to some implementations is shown. In some implementations, example operation 500 may be provided by a speech enhancement system (such as...) Figure 1 Voice enhancement component 120 or Figure 2 The speech enhancement system 200 is executed.

[0059] The speech enhancement system receives a series of frames of an audio signal (510). In some aspects, the audio signal may represent a single channel of audio data. The speech enhancement system denoises the first frame in the series of frames based at least in part on the temporal correlation between the frames (520). In some implementations, the first frame may be denoised based on an MF-MVDR beamformer that reduces the power of the noise components of the audio signal without distorting the speech components.

[0060] The speech enhancement system infers the speech probability associated with the denoised first frame based on a neural network model (530). The speech enhancement system further generates a first speech signal and a first noise signal based on the speech probability associated with the denoised first frame, wherein the first speech signal and the first noise signal represent the speech component and noise component of the audio signal in the first frame, respectively (540). In some implementations, the speech component of the audio signal in the denoised first frame may be equal to the speech component of the audio signal in the first speech signal.

[0061] The speech enhancement system determines a first spectral suppression gain (550) based on a first speech signal and a first noise signal. In some implementations, determining the first spectral suppression gain may include: determining, at least in part, the number (M) of VAD features indicating the presence of speech in a first frame based on the first speech signal and the first noise signal; determining M speech probabilities associated with the first speech signal based on the M VAD features; and determining the amplitude or power of residual noise in the first speech signal based on the M speech probabilities associated with the first speech signal. The speech enhancement system further suppresses residual noise in the first speech signal based on the first spectral suppression gain (560).

[0062] In some implementations, the number of VAD features may be greater than 1 (M>1). In some implementations, the amplitude or power of the residual noise in the first speech signal may be determined solely based on the lowest speech probability among the M speech probabilities associated with the first speech signal. In some implementations, each of the M speech probabilities associated with the first speech signal is determined based on the corresponding GMM. In some implementations, the M VAD features may include the normalized difference between the first speech signal and the first noise signal. In some implementations, the M VAD features may include at least one of cepstral peaks, spectral entropy, or harmonic product spectrum (HPS) associated with the first speech signal.

[0063] In some aspects, the speech enhancement system may further determine an IFC vector associated with the speech components of an audio signal based at least in part on the speech probabilities associated with the denoised first frame; denoise a second frame in a series of frames based at least in part on the IFC vector; infer the speech probabilities associated with the denoised second frame based on a neural network model; generate a second speech signal and a second noise signal based on the speech probabilities associated with the denoised second frame, wherein the second speech signal and the second noise signal represent the speech components and noise components of the audio signal in the second frame, respectively; determine a second spectral suppression gain based on the second speech signal and the second noise signal; and suppress residual noise in the second speech signal based on the second spectral suppression gain.

[0064] Those skilled in the art will recognize that information and signals can be represented using any of a wide variety of different techniques and skills. For example, data, instructions, commands, information, signals, bits, symbols, and chips, which can be referenced throughout the description above, can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0065] Furthermore, those skilled in the art will recognize that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure.

[0066] The methods, sequences, or algorithms described in conjunction with the aspects disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor.

[0067] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader scope of this disclosure as set forth in the appended claims. Accordingly, the description and drawings are to be viewed in an illustrative rather than a limiting sense.

Claims

1. A method of speech enhancement, comprising: receiving a series of frames of an audio signal; denoising a first frame of the series of frames based at least in part on a temporal correlation between the series of frames; inferring, based on a neural network model, a speech probability associated with the denoised first frame; generating, based on the speech probability associated with the denoised first frame, a first speech signal and a first noise signal, the first speech signal and the first noise signal respectively representing a speech component and a noise component of the audio signal in the first frame; determining, based on the first speech signal and the first noise signal, a first spectral suppression gain; and suppressing, based on the first spectral suppression gain, residual noise in the first speech signal.

2. The method of claim 1, wherein, the audio signal comprises a single channel of audio data.

3. The method of claim 1, wherein, the first frame is denoised based on a multi-frame minimum variance distortionless response (MF-MVDR) beamformer that reduces power of the noise component of the audio signal without distorting the speech component.

4. The method of claim 1, further comprising: determining, based at least in part on the speech probability associated with the denoised first frame, an inter-frame correlation (IFC) vector associated with a speech component of the audio signal; denoising a second frame of the series of frames based at least in part on the IFC vector; inferring, based on the neural network model, a speech probability associated with the denoised second frame; generating, based on the speech probability associated with the denoised second frame, a second speech signal and a second noise signal, the second speech signal and the second noise signal respectively representing a speech component and a noise component of the audio signal in the second frame; determining, based on the second speech signal and the second noise signal, a second spectral suppression gain; and suppressing, based on the second spectral suppression gain, residual noise in the second speech signal.

5. The method of claim 1, wherein, the speech component of the audio signal in the denoised first frame is equal to the speech component of the audio signal in the first speech signal.

6. The method of claim 1, wherein, the determination of the first spectral suppression gain comprises: determining, based at least in part on the first speech signal and the first noise signal, a number (M) of voice activity detection (VAD) features that indicate whether speech is present in the first frame; determining, based on the M VAD features, M speech probabilities respectively associated with the first speech signal; and determining, based on the M speech probabilities respectively associated with the first speech signal, a magnitude or power of the residual noise in the first speech signal.

7. The method of claim 6, wherein, the magnitude or power of the residual noise in the first speech signal is determined based only on a lowest speech probability of the M speech probabilities respectively associated with the first speech signal.

8. The method of claim 6, wherein, each of the M speech probabilities respectively associated with the first speech signal is determined based on a respective Gaussian mixture model (GMM).

9. The method of claim 6, wherein, M>1。 10. The method of claim 6, wherein, the M VAD features comprise a normalized difference between the first speech signal and the first noise signal.

11. The method of claim 6, wherein, The M VAD features include at least one of a cepstral peak, a spectral entropy, or a harmonic product spectrum (HPS) associated with the first speech signal.

12. A speech enhancement system comprising: a processing system; and a memory storing instructions that, when executed by the processing system, cause the speech enhancement system to: receive a series of frames of an audio signal; denoise a first frame of the series of frames based at least in part on a temporal correlation between the series of frames; inference, based on a neural network model, a speech probability associated with the denoised first frame; generate a first speech signal and a first noise signal based on the speech probability associated with the denoised first frame, the first speech signal and the first noise signal respectively representing a speech component and a noise component of the audio signal in the first frame; determine a first spectral suppression gain based on the first speech signal and the first noise signal; and suppress residual noise in the first speech signal based on the first spectral suppression gain.

13. The speech enhancement system of claim 12, wherein, The audio signal includes a single channel of audio data.

14. The speech enhancement system of claim 12, wherein, The first frame is denoised based on a multi-frame minimum variance distortionless response (MF-MVDR) beamformer that reduces a power of the noise component of the audio signal without distorting the speech component.

15. The speech enhancement system of claim 12, wherein, Execution of the instructions further causes the speech enhancement system to: determine, based at least in part on the speech probability associated with the denoised first frame, an inter-frame correlation (IFC) vector associated with a speech component of the audio signal; denoise a second frame of the series of frames based at least in part on the IFC vector; inference, based on the neural network model, a speech probability associated with the denoised second frame; generate a second speech signal and a second noise signal based on the speech probability associated with the denoised second frame, the second speech signal and the second noise signal respectively representing a speech component and a noise component of the audio signal in the second frame; determine a second spectral suppression gain based on the second speech signal and the second noise signal; and suppress residual noise in the second speech signal based on the second spectral suppression gain.

16. The speech enhancement system of claim 12, wherein, The speech component of the audio signal in the denoised first frame is equal to the speech component of the audio signal in the first speech signal.

17. The speech enhancement system of claim 12, wherein, The determination of the first spectral suppression gain includes: determining, based at least in part on the first speech signal and the first noise signal, a number (M) of voice activity detection (VAD) features that indicate whether speech is present in the first frame; determining, based on the M VAD features, M speech probabilities respectively associated with the first speech signal; and determining, based on the M speech probabilities respectively associated with the first speech signal, a magnitude or a power of the residual noise in the first speech signal.

18. The speech enhancement system of claim 17, wherein, Each of the M speech probabilities respectively associated with the first speech signal is determined based on a respective Gaussian mixture model (GMM).

19. The speech enhancement system of claim 17, wherein, M>1。 20. The speech enhancement system of claim 17, wherein, The M VAD features include at least one of a normalized difference between the first speech signal and the first noise signal, a cepstral peak of the first speech signal, a spectral entropy of the first speech signal, or a harmonic product spectrum (HPS) associated with the first speech signal. The M VAD features include at least one of a normalized difference between the first speech signal and the first noise signal, a cepstral peak of the first speech signal, a spectral entropy of the first speech signal, or a harmonic product spectrum (HPS) associated with the first speech signal.

Images