Machine learning-assisted spatial noise estimation and suppression

Abstract

This invention provides a system and method for estimating and suppressing spatial noise using machine learning support. [Solution] The noise estimation and suppression method includes the steps of: estimating the probability of speech and noise for each band of the input audio signal using a machine learning classifier; estimating a set of averages of speech and noise or a set of averages and covariances of speech and noise based on the probability and microphone covariance over the bands using a directional model; estimating the average and covariance of noise power based on the probability and power spectrum using a level model; determining a first noise suppression gain based on the directional model; determining a second noise suppression gain based on the level model; selecting the first noise suppression gain or the second noise suppression gain or their sum based on the signal-to-noise ratio of the input audio signal; and scaling the time-frequency representation of the input signal by the selected noise suppression gain.

Description

[Technical Field]

[0001]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Application No. 63 / 110,228, filed on November 5, 2020, and U.S. Provisional Application No. 63 / 210,215, filed on June 14, 2021, both of which are invoked in their entirety by reference.

[0002]

[0002] Technical field The disclosures in this case generally relate to audio signal processing, and in particular to noise estimation and suppression in voice communications. [Background technology]

[0003]

[0003] Voice communication noise suppression algorithms are effectively implemented in edge devices such as telephones, laptops, and conference systems. A common problem with bidirectional voice communication is that background noise at each user's location is transmitted along with the user's voice signal. If the signal-to-noise ratio (SNR) of the integrated signal received by the edge device is too low, the clarity of the reconstructed voice will be degraded, resulting in a poor user experience. [Overview of the project]

[0004]

[0004] Implementations for spatial noise estimation and suppression using machine learning assistance will be described. In some embodiments, the audio processing method is: A step of receiving the bands of the power spectrum of an input audio signal and the microphone covariance of each band, wherein the microphone covariance is based on the arrangement of the microphones used to capture the input audio signal; About each band: Using a machine learning classifier, estimate the probabilities of speech and noise; Using a directional model, estimate the set of speech and noise means, or the set of speech and noise means and covariances, based on the microphone covariance for the probability and band; Using a level model, the mean and covariance of noise power are estimated based on probability and power spectrum; The first noise suppression gain is determined based on the first output of the directional model; The second noise suppression gain is determined based on the second output of the level model; A step of selecting one of a first noise suppression gain, a second noise suppression gain, or the sum of the first and second noise suppression gains, based on the signal-to-noise ratio of the input audio signal; The time-frequency representation of the input signal is scaled by a first or second noise suppression gain selected for the band; and This includes the step of converting the time-frequency representation into an output audio signal.

[0005]

[0005] In some embodiments, the method further includes: receiving an input audio signal containing multiple blocks / frames using at least one processor; About each block / frame: The process includes the steps of: using at least one processor to convert a block / frame into multiple subbands (each subband having a spectrum different from the others); using at least one processor to combine the subbands into a band; and using at least one processor to determine the banded power.

[0006]

[0006] In some embodiments, the machine learning classifier is a neural network comprising an input layer, an output layer, and one or more hidden layers. For example, the neural network is a deep neural network comprising three or more layers, preferably more than three.

[0007]

[0007] In some embodiments, the microphone covariance is represented as a normalized vector.

[0008]

[0008] In some embodiments, the method further includes the steps of determining the first noise suppression gain: calculating the probability of speech for a band; setting the first noise suppression gain to be equal to the maximum suppression gain if the probability of speech for a band is less than a threshold; and setting the first noise suppression gain based on a gain ramp if the calculated probability of speech for a band is greater than a threshold.

[0009]

[0009] In some embodiments, the probability of speech is calculated using a set of mean covariances, speech, and noise estimated by a directional model.

[0010]

[0010] In some embodiments, the probability of speech is calculated using a covariance vector estimated by a directional model, a set of mean speech and noise, and a multi-variable, joint Gaussian density function.

[0011]

[0011] In some embodiments, the method further includes: the step of determining a second noise suppression gain further includes: setting the second noise suppression gain to be equal to the maximum suppression gain if the band power is less than a first threshold; setting the second noise suppression gain to zero if the band power is between the first threshold and the second threshold (where the second threshold is higher than the first threshold); and setting the second noise suppression gain based on a gain ramp if the band power is higher than the second threshold.

[0012]

[0012] In some embodiments, the estimation step using a directional model utilizes time-frequency tiles that have been classified as speech and noise, with those classified as reverberation excluded.

[0013]

[0013] In some embodiments, the method further includes: estimating the mean of the speech based on the speech probability and microphone covariance for the band using a directional model or level model; and further including: calculating a time-averaged estimate of the mean of the speech by weighting the filter input by the speech probability using a first-order low-pass filter with the microphone covariance vector and the mean of the speech as filter inputs.

[0014]

[0014] In some embodiments, the method further includes: estimating the mean of the noise based on the probability of the noise and the microphone covariance over the band using a directional model or a level model; and further including: calculating a time-averaged estimate of the mean of the noise by weighting the filter input by the probability of the noise using a first-order low-pass filter with the microphone covariance vector and the mean of the noise as filter inputs.

[0015]

[0015] In some embodiments, the method further includes: estimating the speech covariance based on the speech probability and microphone covariance for the band using a directional model or level model; and further including: calculating a time-averaged estimate of the speech covariance by weighting the filter input by the speech probability using a first-order low-pass filter with the microphone covariance vector and the speech covariance as filter inputs.

[0016]

[0016] In some embodiments, the method further includes: estimating the covariance of the noise based on the probability of the noise and the microphone covariance for the band using a directivity model or a level model; further including: using a first-order low-pass filter with the microphone covariance vector and the covariance of the speech as filter inputs, and weighting the filter inputs by the probability of the noise to calculate a time-averaged estimate of the covariance of the speech.

[0017]

[0017] In some embodiments, the system includes at least one computer processor and a non-transitory computer-readable storage medium storing instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform any of the above methods.

[0018]

[0018] In some embodiments, the non-transitory computer-readable medium stores instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform any of the above methods.

[0019]

[0019] Other implementations disclosed herein are directed to systems, devices, and computer-readable media. Details of the disclosed implementations are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the specification, drawings, and claims.

[0020]

[0020] The specific implementations disclosed herein offer one or more of the following advantages: The disclosed embodiments use directionality and machine learning (e.g., neural networks) to bring low-cost, high-quality noise estimation and suppression to voice communication applications. The disclosed embodiments for noise estimation and suppression can be implemented in a variety of edge devices and do not require multiple microphones. By using neural networks, scaling is possible to large variations and variability in background noise. [Brief explanation of the drawing]

[0021]

[0021] The drawings show the specific arrangement and order of schematic elements, such as those representing devices, units, instruction blocks, and data elements, for the sake of simplicity of explanation. However, those skilled in the art will understand that the specific arrangement and order of schematic elements in the drawings are not intended to indicate that a particular order or sequence of processing, or division of processes, is mandatory. Furthermore, the inclusion of schematic elements in the drawings is not intended to indicate that such elements are mandatory in all embodiments, or that features represented by such elements may not be included in or combined with other elements in some implementations.

[0022]

[0022] Furthermore, connecting elements such as solid lines, dashed lines, or arrows are used in drawings to indicate connections, relationships, or associations between or within two or more other schematic elements, and the absence of any such connecting elements in a drawing is not intended to indicate that a connection, relationship, or association cannot exist. In other words, any connection, relationship, or association between elements may not be shown in a drawing so as not to obscure the disclosure. Furthermore, for the sake of clarity of explanation, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of signals, data, or instructions, a person skilled in the art will understand that such an element may, as necessary, represent one or more signal paths affecting the communication. [Figure 1]

[0023] Figure 1 is a block diagram of a machine learning-assisted spatial noise estimation and suppression system according to one embodiment. [Figure 2]

[0024] Figure 2 illustrates noise suppression gain calculation based on a level model according to some embodiments. [Figure 3]

[0025] Figure 3 illustrates the noise suppression gain calculation based on a directional model according to some embodiments. [Figure 4A]

[0026] Figure 4A shows a flowchart of the noise estimation and suppression process in voice communication using directionality and machine learning, according to some embodiments. [Figure 4B]

[0026] Figure 4B shows a flowchart of the noise estimation and suppression process in voice communication using directionality and machine learning according to some embodiments. [Figure 5]

[0027] Figure 5 is a block diagram of a system for implementing the features and processes described in relation to Figures 1-4, according to some embodiments.

[0028] The same reference symbols used in various drawings represent similar elements. [Modes for carrying out the invention]

[0023]

[0029] The following detailed description includes numerous specific details to provide a thorough understanding of the various embodiments described. It will be apparent to those skilled in the art that the various implementations described may be realized without these specific details. In other examples, well-known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure the aspects of the embodiments. Several features are described below, each of which can be used independently or in some combination with other features.

[0024]

[0030] terminology system Where used herein, the term “include” and its variations should be understood as open-ended terms meaning “includes, but not limited to.” The term “or” should be understood as “and / or” unless the context clearly indicates otherwise. The term “based on” should be understood as “at least partially based on.” The terms “a certain exemplary implementation” and “exemplary implementation” should be understood as “at least one exemplary implementation.” The term “another implementation” should be understood as “at least one other implementation.” The terms “determined,” “determines,” or “determining” should be understood as obtaining, receiving, operating, calculating, estimating, predicting, or deriving. Furthermore, in the following descriptions and claims, unless otherwise specified, all technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art to which this disclosure belongs.

[0025]

[0031] overview Traditional noise suppression solutions use two or more microphones to capture background noise, with one microphone close to the user's mouth and the other further away. The signals from the two microphones are subtracted to remove background noise common to both signals. However, this technique does not work for edge devices with only one microphone, or, in the case of mobile phones, when the user shakes or rotates the phone while speaking. Other noise suppression algorithms attempt to adapt by continuously detecting noise patterns in the audio signal and processing the speech frame by frame or block by block, where each block may contain two or more frames. While these existing adaptive algorithms work well in specific use cases, they do not scale well to background noise with high diversity and variability.

[0026]

[0032] Recently, deep neural networks have been used to suppress noise in voice communications. However, these solutions require considerable computing power and are difficult to implement in real-time communication systems. The disclosed embodiments utilize a combination of directionality and machine learning (e.g., deep neural networks) to provide low-cost, high-quality noise estimation and suppression for voice communication applications.

[0027]

[0033] Figure 1 is a block diagram of a machine learning-assisted spatial noise estimation and suppression system 100 according to one embodiment. The system 100 includes a filter bank 101, a banding unit 102, a machine learning classifier 103 (e.g., a neural network such as a deep neural network (DNN)), a direction detection unit 104, a speech / noise directivity model 105, a noise level model 106, a noise suppression gain unit 107, a multiplication unit 108, and an inverse filter bank 109. In one embodiment, the filter bank 101 (e.g., a short-time Fourier transform (STFT)) receives a time-domain input audio signal and transforms the time-domain input audio signal into multiple subbands, each subband having a different frequency spectrum (e.g., they are time / frequency tiles). The subbands of each block / frame are input to the banding unit 102, which combines the block / frame subbands into bands according to a psychoacoustic model and outputs banded power 110 (e.g., banded power expressed in dB). The subbands output by the filter bank 101 are also input to the direction detection unit 104, which generates and outputs a microphone covariance vector 112. In some embodiments, there is one covariance vector for each band, and a common banding matrix is ​​used in the banding unit 102 and the direction detection unit 104. In some implementations, the banding unit 102 and the direction detection unit 104 are a single unit that generates the band power in decibels and the covariance vector for each band.

[0028]

[0034] The band power 110 is input to the machine learning classifier 103. In embodiments, the machine learning classifier 103 is a pre-trained neural network classifier that estimates and outputs probabilities 111 for multiple classes, the classes including, but not limited to, speech classes, static noise classes, non-static noise classes, and reverberation classes for each block / frame and each band. In some embodiments, the static noise class is used to operate the speech / noise directivity model 105 and the noise level model 106.

[0029]

[0035] In the following disclosure, it is assumed that the machine learning classifier estimates and outputs the probabilities of speech and static noise. In embodiments where the machine learning classifier outputs the probability of reverberation, the estimated probabilities of speech, static noise, non-static noise, and reverberation are used to separate reverberation from noise. In some embodiments, no time-frequency tile classified as reverberation is added to the speech / noise directivity model 105 or the noise level model 106.

[0030]

[0036] The probability 111 is input to the speech / noise directivity model 105 along with the microphone covariance vector 112. The probability 111 is also input to the speech / noise level model 106 along with the banded power 110. For each band and each block / frame, the speech / noise directivity model 105 estimates and outputs the respective mean and / or covariance 114 for speech and noise in each band, based on the microphone covariance vector 112. The noise level model 106 estimates and outputs the mean and variance 113 of the noise power in each band for each block / frame. In some embodiments, the noise level model 106 also outputs the mean and variance of speech.

[0031]

[0037] The noise suppression gain unit 107 receives the outputs 113, 114 of the speech / noise directivity model 105 and the noise level model 106, the banded power 110, and the microphone covariance vector 112, respectively. The noise suppression gain unit 107 calculates and outputs the suppression gain used by the multiplication unit 108, and scales the output subbands by the filter bank 101 to suppress noise in each subband. The subbands are then converted into time-domain output audio signals by the inverse filter bank 109. The components of system 100 are described in more detail below. In some embodiments, the output of the machine learning classifier 103 is used either directly as noise gain or to calculate the noise suppression gain.

[0032]

[0038] Machine learning classifier As described above, the machine learning classifier 103 takes banded power 110 as input and provides as output the speech and noise probabilities 111 given as follows:

[0033]

number

[0034]

[0039] In the illustrated example, the machine learning classifier 103 includes a neural network (NN). The input to the exemplary NN 103 includes 61 banded powers of the current frame, and an input linear dense layer maps the banded powers to 256 features. The 256 features pass through a set of GRU layers, each containing 256 hidden units, and finally reach a dense layer with several nonlinear kernels. NN 103 outputs relative weights for each class and each band, which are converted to probabilities using a softmax function. NN 103 is trained using noisy, labeled speech and cross-entropy as the cost function. In this embodiment, the Adam optimizer is used to train the neural network. Other examples of machine learning classifiers include, but are not limited to, k-nearest neighbors, support vector machines, or decision trees.

[0035]

[0040] In this embodiment, NN 103 can be trained as follows: 1. Obtain a set of close-mic recorded dry, clean speech. For example, VCTK (Centre for Speech Technology Voice Cloning Toolkit) may be used; 2. Obtain a set of noise data. For example, AudioSet data available at http: / / research.google.com / audioset / can be used; and 3. Before the training process begins, features of the speech and noise data are extracted. For example, 61 band energies (in dB) are calculated at a rate of, for example, 50 Hz for a band interval between 50 Hz and 8000 Hz.

[0036] 4. As part of the feature extraction process: a. Determine the measure of speech power for each speech vector. For example, speech files are passed through a voice activity detector (VAD), and the mean A-weighted power for all speech passing through the VAD is calculated (e.g., dBFS); and b. Determine the measured noise power for each noise vector. For example, the average A-weighted power (e.g., dBFS) is calculated over the vectors.

[0037] 5. Regarding each speech vector in each training epoch: a. Select a random noise vector from the noise dataset associated with the speech; b. Derive a random SNR by mixing speech and noise. For example, this can be done from a normal distribution where the mean is, for example, 20 dB SNR and the standard deviation is, for example, 10 dB SNR; c. Determine the gain (dB) for mixing speech and noise from the selected SNR, the predetermined speech power (S), and the predetermined noise power (N) according to gain = SNR - S + N; d. Apply reverb to the speech and record it with significant reverb on time-frequency tiles; e. Mix the reverberated speech and noise using the given gain. Since the features are on a dB scale, the "mix" can be approximated by taking the max() of the reverberated speech power and the gain-applied noise power at each time-frequency tile. While doing so, record which time-frequency tile noise ultimately becomes dominant over the reverberated speech. The final training vector presented to the NN during training consists of 61 band energies per frame, along with a ground truth class label based on whether noise, speech, or reverberation was dominant at each time-frequency tile during the above procedure; and f. The training vector is shown to the network, and the cross-entropy loss is reduced relative to the ground truth calculated during the above procedure.

[0038]

[0041] Microphone covariance matrix In some embodiments, the filter bank 101 is implemented using the Short-Time Fourier Transform (STFT). t Let (k) be the k-th bin data (subband) of the STFT of block / frame t for all microphone inputs. t (k) is a vector of length N, where N is the number of microphones, M is the number of subbands, and α is a weighting factor that weights the contributions of past and estimated covariances. In real-time audio processing, the microphone covariance of band f is calculated as follows:

[0039]

number

[0040]

[0042] It should be noted that equations [2] and [3] hold for "rectangular banding," where each subband contributes exactly one output band with a gain of 1. In general, some weight w within the range [0,1] kf There exists a function that indicates how much the input subband k contributes to the output band f. For each subband k, all weights w kf The sum across all bands f must be 1. In this way, it is possible to perform banding of any shape. For example, linear, log, or cosine or triangular banding at Mel frequencies can be used in addition to rectangular banding.

[0041]

number

[0043] In one embodiment, the banded power can be calculated directly from the filter bank 101, for example, a heterogeneous filter bank which has an integrated banding scheme. In such an embodiment, decimation is not used, and the block rate determines the block frame t.

[0042]

[0044] To simplify the notation, the normalized covariance matrix is ​​a real vector 112v t,f This arrangement is used because the covariance matrix is ​​Hermitian, and its diagonal elements are real numbers. The normalized covariance is obtained by dividing the covariance matrix by its trace, and as a result, it represents only the direction, with any level components removed.

[0043]

[0045] The elements of this covariance matrix are c m,n If expressed as such, the rearranged vector for a 3-microphone system is given as follows, where m and n are indices of the covariance matrix elements:

[0044]

number

[0046] In some embodiments, v t,f is normalized. Systems with more or fewer microphones will have more or fewer elements, and thus the system is scalable to systems with any number of microphones.

[0045]

[0047] Speech / Noise Directional Model In some embodiments, the speech / noise directivity model 105 receives, as inputs, the covariance vectors (one per band) and the probabilities 111 output by the machine learning classifier 103, and v t,f estimates the mean and / or covariance matrix of v. t,f There are at least two embodiments for estimating the mean and / or covariance matrix of v, which are described as follows: 1. Estimate only the means of speech and noise and use the estimated means as the directivity model:

[0046]

Number

[0048] In equations [7] and [8], μ n (t,f) and μ s (t,f) are the directivity models for noise and speech, respectively, for block / frame t and frequency band f. These are the means of the normalized microphone covariance matrices. Note that w, used in equations [7] and [8] and hereafter, is a weighting factor that controls the length of the time averaging window and may be different for speech and noise or for mean and variance.

[0047] 2. The mean and covariance vectors of noise and speech define a spatial model, which can be calculated as follows:

[0048]

Number

[0049] Noise Level Model The Noise Level Model 106 is a banded power 110 (L t,f The input is ) in dB, and the mean and variance of the noise for band f and block / frame t are estimated:

[0049]

number

[0050] Note that in equations [7]-

[14] , time-averaging is used to estimate the mean and covariance of the random variable (either level or direction), under the assumption that the stochastic process is ergodic and static over a certain period of time. The mean is achieved using a first-order low-pass filter model, in which, for each frame, the input to the low-pass filter (mean or covariance) is weighted by the probability of speech or noise.

[0050]

[0051] Calculation of noise suppression gain based on level models Figure 2 shows the noise suppression gain G based on the noise level model 106 in some embodiments. L This figure shows the calculation of (b,f). The vertical axis represents the gain in dB, and the horizontal axis represents the level in dB. In Figure 2, G0 is the maximum suppression gain, and the gain ramp slope β and k are tuning parameters:

[0051]

number

[0052] If the current signal-to-noise level is greater than a predefined threshold for the signal-to-noise ratio (see equation

[19] ), the noise suppression gain is G L Otherwise, the speech / noise directivity model 105 is used to calculate Gs. The system calculates the probability of speech for band f and block / frame t using at least one of two methods: 1. Normalized microphone covariance vector v t,fUsing the mean and speech / noise directivity model 105, calculate the probability of speech:

[0052]

number

[0053]

number

[0053] Calculation of noise suppression gain based on a directional model Figure 3 shows the calculation of the noise suppression gain based on the speech / noise directivity model 105. The vertical axis represents the gain in dB, the horizontal axis represents the speech probability, and the noise suppression gain is given as follows:

[0054]

number

[0055]

[0054] The final suppression gain G(t,f) is calculated for each band f and each block / frame t (for example, by the noise suppression gain unit 107) as follows:

[0056]

number

[0057]

[0055] In some embodiments, obtaining the estimated SNR in

[19] involves using the output of the VAD (Voice activity detector) to drive an automatic gain control (AGC) component (not shown) to make the speech level equivalent to a predefined power level (dB), and from that predefined power level, the estimated noise level μ L This can be done by subtracting (t,f) and estimating the speech level using a method similar to that used for noise estimation:

[0058]

number

[0059]

number

[0056] The final suppression gain G(t,f) is used in the multiplication unit 108 to scale the subbands output by the filter bank 101 and suppress noise in each subband. For example, the gain G(t,f) is applied to all subbands k belonging to band f:

[0060]

number

[0061]

[0057] The output of the multiplication unit 108 is then converted by the inverse filter bank 109 into a time-domain output audio signal, resulting in a noise-suppressed output signal.

[0062]

[0058] Process Example Figures 4A and 4B are flowcharts of a process 400 for noise estimation and suppression in voice communications using directional and deep neural networks, according to some embodiments. Process 400 can be implemented using system 500 shown in Figure 5.

[0063]

[0059] Process 400 begins by receiving an input audio signal containing a number of blocks / frames (401). For each block / frame, Process 400 proceeds to convert the block / frame into subbands (where each subband has a different spectrum from the other subbands) (402), combine the subbands into bands to determine the power of each band (403), and determine the microphone covariance based on the subbands (404).

[0064]

[0060] Process 400 continues by using a machine learning classifier (e.g., a neural network) to estimate the probabilities of speech and noise for each band and each block / frame (405), using a directional model to estimate a set of mean speech and noise, or a set of mean speech and noise and covariance, based on microphone covariance and probability (406), using a level model to estimate the mean and variance of noise power based on probability and band power (407), calculating a first noise suppression gain based on a first output of the directional model (408), determining a second noise suppression gain based on a second output of the level model (409), and selecting one of the first noise suppression gain, or the second noise suppression gain, or the sum of the first and second noise suppression gains, based on the signal-to-noise ratio of the input audio signal (410).

[0065]

[0061] Process 400 then scales each subband of each band by a first or second noise suppression gain selected for the band (411), and converts the scaled subbands into an output audio signal (412).

[0066]

[0062] Example of System Architecture Figure 5 shows a block diagram of an exemplary system for implementing the features and processes described in relation to Figures 1-5 according to an embodiment. System 500 includes any device capable of playing audio, which includes, but is not limited to, a smartphone, tablet computer, wearable computer, vehicle computer, game console, surround sound system, or kiosk.

[0067]

[0063] As shown in the figure, the system 500 includes a central processing unit (CPU) 501, which can perform various operations according to programs stored in, for example, read-only memory (ROM) 502, or programs loaded into random access memory (RAM) 503 from a memory unit 508. The RAM 503 also stores data necessary for the CPU 501 to perform various operations as needed. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 509. An input / output (I / O) interface 505 is also connected to a bus 504.

[0068]

[0064] The following components are connected to the I / O interface 505: an input unit 506 which may include a keyboard, mouse, etc.; an output unit 507 which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 508 which includes a hard disk, or any other suitable storage device; and a communication unit 509 which includes a network interface card such as a network card (e.g., wired or wireless).

[0069]

[0065] In some implementations, the input unit 506 includes one or more microphones located in different positions (depending on the host device) to enable the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, or other appropriate formats).

[0070]

[0066] In some implementations, the output unit 507 includes a system with a varying number of speakers. As shown in Figure 5, the output unit 507 can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, or other appropriate formats) depending on the capabilities of the host device.

[0071]

[0067] The communication unit 509 is configured to communicate with other devices (for example, via a network). If necessary, the drive 510 is also connected to the I / O interface 505. The drive 510 is equipped with removable media 511 such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or other suitable removable media, and as a result, computer programs read from there are installed on the storage device 508 as necessary. Although the system 500 is described to include the above components, it is possible to add, remove, and / or replace some of these components in actual applications, and all such changes or substitutions are all within the scope of the present disclosure, as will be understood by those skilled in the art.

[0072]

[0068] The embodiments of the systems described herein can be implemented in a suitable computer-based sound processing network environment for processing digital or digitized audio files. The adaptive audio system portion may include one or more networks containing any desired number of individual machines and one or more routers (not shown) that play a role in buffering and routing data transmitted between computers. Such networks may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0073]

[0069] In accordance with exemplary embodiments of the present disclosure, the above process may be implemented as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product which includes a computer program tangibly embodied on a machine-readable medium, and a computer program which includes program code for performing a method. In such embodiments, the computer program may be downloaded and installed from a network via a communication unit 509 and / or installed from removable media 511, as shown in Figure 5.

[0074]

[0070] In general, various exemplary embodiments of the present disclosure can be implemented in hardware or special-purpose circuits (e.g., control circuits), software, logic, or any combination thereof. For example, the units discussed above can be performed by a control circuit (e.g., a CPU in combination with other components in Figure 5), and thus the control circuit can perform the operations described in the present disclosure. Some embodiments can be implemented in hardware, while others can be implemented in firmware or software, which can be performed by a controller, microprocessor, or other computing device (e.g., a control circuit). Various aspects of the exemplary embodiments of the present disclosure have been described and explained using block diagrams, flowcharts, or any other graphical representation, but it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, in non-limiting examples, in hardware, software, firmware, special-purpose circuits or logic, general-purpose hardware or controllers or other computing devices, or any combination thereof.

[0075]

[0071] Furthermore, the various blocks shown in the flowchart can be viewed as method steps and / or operations resulting from the operation of computer program code and / or as a plurality of combined logic circuit elements constructed to perform related functions. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, and a computer program including program code configured to perform the above method.

[0076]

[0072] In the context of this disclosure, machine-readable media may be any tangible medium that contains or can store a program for use by or related to an instruction execution system, apparatus, or device. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may be non-transient and may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, devices, or any suitable combination thereof. More specific examples of machine-readable storage media include electrical connections having one or more wires, portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0077]

[0073] Computer program code for performing the methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes can be provided to a general-purpose computer processor, a dedicated computer, or other programmable data processing device having a control circuit, so that when the program code is executed by the computer processor or other data processing device, it causes the functions / operations specified in the flowchart and / or block diagram to be performed. The program code may be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server, or distributed across one or more remote computers and / or servers.

[0078]

[0074] Although this specification includes many specific implementation details, these should not be interpreted as limitations on the scope of what can be claimed, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in this specification in the context of individual embodiments may also be realized in combinations in a single embodiment. Conversely, various functions described in the context of a single embodiment may also be realized individually in multiple embodiments or in any suitable sub-combinations. Furthermore, although features are described above as operating in a particular combination and may even be initially claimed as such, in some cases one or more features in a claimed combination may be extracted from the combination, and the claimed combination may cover a sub-combination or a variation of a sub-combination. The illustrated logical flow does not require a specific order or sequence shown in order to obtain the desired result. Furthermore, other steps may be provided, steps may be removed from the described flow, and other components may be added to or removed from the described system. Accordingly, other implementations are within the scope of the following claims. [Note] (Note 1) Audio processing method: A step of receiving the bands of the power spectrum of an input audio signal and the microphone covariance of each band using at least one processor, wherein the microphone covariance is based on the arrangement of microphones used to capture the input audio signal; About each band: Using a machine learning classifier, estimate the probabilities of speech and noise; Using a directional model, estimate a set of speech and noise means, or a set of speech and noise means and covariances, based on the probabilities and the microphone covariance for the band; Using a level model, the mean and covariance of the noise power are estimated based on the probability and the power spectrum; Using the at least one processor, a first noise suppression gain is determined based on the first output of the directional model; Using at least one of the processors, a second noise suppression gain is determined based on the second output of the level model; Using at least one of the above-mentioned processors, one of the first noise suppression gain, the second noise suppression gain, or the sum of the first and second noise suppression gains is selected based on the signal-to-noise ratio of the input audio signal; Using at least one of the processors, the time-frequency representation of the input audio signal is scaled by a first or second noise suppression gain selected for the band; and A step of converting the time-frequency representation into an output audio signal using at least one of the aforementioned processors; A method that includes this. (Note 2) In the method described in Appendix 1, further: A step of receiving an input audio signal containing multiple blocks / frames using at least one of the aforementioned processors; About each block / frame: Using at least one of the aforementioned processors, the block / frame is converted into a plurality of subbands, each subband having a spectrum different from that of the other subbands; Using the at least one processor, combine the subbands into the bands; and A step of determining the banded power using at least one of the aforementioned processors; A method that includes this. (Note 3) The method described in Appendix 1, wherein the machine learning classifier is a neural network. (Note 4) A method according to Appendix 1 or 2, wherein the microphone covariance is expressed as a normalized vector. (Note 5) In the method described in any one of the appendices 1-4, the step of determining the first noise suppression gain is: Calculate the probability of speech for the aforementioned band; If the probability of speech for the band is less than a threshold, the first noise suppression gain is set to be equal to the maximum suppression gain; and To increase the probability of speech, the first noise suppression gain is set by ramping from the maximum suppression gain toward zero; Methods that include... (Note 6) The method described in Appendix 5, wherein the probability of the speech is calculated using the set of means of the covariance, speech, and noise estimated by the directional model. (Note 7) The method described in Appendix 5, wherein the probability of the speech is calculated using the set of means of the covariance, speech, and noise estimated by the directional model, and a multivariable simultaneous Gaussian density function. (Note 8) In the method described in any one of the appendices 1-7, the step of determining the second noise suppression gain is: If the power of the band is less than the first threshold, the second noise suppression gain is set to be equal to the maximum suppression gain; and To increase the probability of speech, the noise suppression gain is set by ramping from the maximum suppression gain toward zero; Methods that include... (Note 9) A method according to any one of the appendices 1-8, wherein the step of estimation using the directional model utilizes time-frequency tiles that have been classified as speech and noise, excluding those classified as reverberation. (Note 10) In the method described in any one of the appendices 1-9, the step of estimating the mean of a speech based on the probability of the speech and the microphone covariance for the band using a directional model or a level model is further: A first-order low-pass filter is used, with the microphone covariance and the mean of the speech as filter inputs, to calculate a time-averaged estimate of the mean of the speech, and the filter inputs are weighted by the probability of the speech; Methods that include... (Note 11) In the method described in any one of the appendices 1-10, the step of estimating the mean of the noise based on the probability of the noise and the microphone covariance for the band using a directional model or a level model is further: A first-order low-pass filter is used, with the microphone covariance vector and the average of the noise as filter inputs. The time-averaged estimation of the average of the noise is calculated, and the filter input is weighted by the probability of the noise. Methods that include... (Note 12) In the method described in any one of the appendices 1-11, the step of estimating the speech covariance based on the speech probability and the microphone covariance for the band using a directional model or a level model is further: A first-order low-pass filter is used, with the microphone covariance vector and the speech covariance as filter inputs, to calculate a time-averaged estimate of the speech covariance, and the filter input is weighted by the probability of the speech. Methods that include... (Note 13) In the method described in any one of the appendices 1-12, the step of estimating the noise covariance based on the probability of the noise and the microphone covariance for the band using a directional model or a level model is further: A first-order low-pass filter is used, with the microphone covariance vector and the noise covariance as filter inputs, to calculate a time-averaged estimate of the noise covariance, and the filter input is weighted by the probability of the noise; Methods that include... (Note 14) At least one computer processor: and A non-temporary, computer-readable storage medium for storing instructions; A system comprising, wherein, when the instruction is executed by the one or more computer processors, the one or more computer processors cause the one or more computer processors to perform the method described in any one of the items 1-13 of the appendix. (Note 15) A non-temporary, computer-readable storage medium for storing instructions, wherein, when executed by one or more computer processors, the instructions cause the one or more computer processors to perform the method described in any one of the items 1-13 of the appendix.

Claims

1. Audio processing method: A step of receiving a representation of an input audio signal using at least one processor; A step of estimating noise suppression gain using a machine learning classifier, wherein the machine learning classifier takes the representation of the input audio signal as input; A step of scaling the time-frequency representation of the input audio signal by the noise suppression gain using at least one of the processors; and A step of converting the time-frequency representation into an output audio signal using at least one of the aforementioned processors; A method that includes this.

2. A method according to claim 1, wherein the representation of the input audio signal includes a banded power spectrum of the input audio signal.

3. The method according to claim 2, further: A step of receiving an input audio signal including multiple blocks using at least one of the aforementioned processors; For each block: Using at least one of the aforementioned processors, the block is transformed into a plurality of subbands, each subband having a spectrum different from that of the other subbands; Using the at least one processor, the subband is coupled to the band; and A step of determining the banded power spectrum using at least one of the aforementioned processors; A method that includes this.

4. A method according to any one of claims 1 to 3, wherein the machine learning classifier is a neural network.

5. The method according to claim 4, wherein the neural network includes an input layer configured to map a representation of the input audio signal to a plurality of features.

6. The method according to claim 5, wherein the neural network further comprises a plurality of layers configured to determine the noise suppression gain based on the plurality of features.

7. A method according to any one of claims 4 to 6, wherein the neural network is trained by a cross-entropy loss function.

8. A machine learning-based classifier for audio processing: A neural network configured to estimate noise suppression gain based on the representation of the input audio signal; The neural network includes: A classifier comprising a plurality of layers, including at least one of an input layer, a GRU layer, and / or a high-density layer.

9. A machine learning-based classifier according to claim 8, wherein the neural network includes the input layer, the input layer is configured to map a representation of the input audio signal to a plurality of features.

10. A machine learning-based classifier according to claim 9, wherein the plurality of layers are configured to determine the noise suppression gain based on the plurality of features.

11. A machine learning-based classifier according to any one of claims 8 to 10, wherein the neural network is trained by a cross-entropy loss function.

12. One or more computer processors: and A non-temporary, computer-readable storage medium for storing instructions; A system comprising, wherein, when the instruction is executed by the one or more computer processors, the system causes the one or more computer processors to perform the method according to any one of claims 1 to 7.

13. A non-temporary, computer-readable storage medium for storing instructions, wherein, when executed by one or more computer processors, the instructions cause the one or more computer processors to perform the method according to any one of claims 1 to 7.

Images