Deep-learning based speech enhancement

Abstract

To provide a system for suppressing noise and enhancing speech and a related method.SOLUTION: A system trains a neural network model that takes banded energies corresponding to an original noisy waveform and produces a speech value indicating an amount of speech present in each band at each frame. The neural network model has a feature extraction block that implements some lookaheads. The feature extraction block is followed by an encoder with steady down-sampling along a frequency domain forming a contracting path. The encoder is followed by a corresponding decoder with steady up-sampling along the frequency domain forming an expanding path. The decoder receives scaled output feature maps from the encoder at a corresponding level. A classification block following the decoder produces a speech value indicating an amount of speech present for each frequency band of a plurality of frequency bands at each frame.SELECTED DRAWING: None

Description

[Technical Field]

[0001] This application claims priority to U.S. Provisional Application No. 63 / 115,213, filed November 18, 2020, U.S. Provisional Application No. 63 / 221,629, filed July 14, 2021, and International Patent Application No. PCT / CN2020 / 124635, filed October 29, 2020, all of which are incorporated herein by reference in their entirety.

[0002] This application relates to noise reduction from speech. More specifically, example embodiments described below relate to applying deep learning models to generate frame-based inferences from large-scale speech contexts. [Background technology]

[0003] The approaches described in this section are approaches that could be advanced, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section constitute prior art by virtue of their inclusion in this section.

[0004] Accurately removing noise from a mixed speech and noise signal is generally difficult given the various forms of speech and types of noise that may be present. Suppressing noise in real time can be particularly challenging. Summary of the Invention

[0005] A system and related methods for suppressing noise and enhancing speech are disclosed. The method includes the steps of: receiving, by a processor, input audio data covering a plurality of frequency bands along a frequency dimension in a plurality of frames along a time dimension; training, by the processor, a neural network model including: a feature extraction block that looks ahead a certain number of frames when extracting features from the input audio data; an encoder including a first series of blocks that generate first feature maps corresponding to increasingly larger receptive fields within the input audio data along the frequency dimension; a decoder including a second series of blocks that receive the output feature map generated by the encoder as an input feature map and generate a second feature map; and a classification block that receives the second feature map and generates, for each frequency band of the plurality of frequency bands in each frame of the plurality of frames, a speech value indicating the amount of speech present; receiving new audio data having one or more frames; running the neural network model on the new audio data to generate new speech values ​​for each frequency band of the plurality of frequency bands in each frame of the one or more frames; generating new output data that suppresses noise in the new audio data based on the new speech values; and transmitting the new output data. [Brief explanation of the drawings]

[0006] Example embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like elements are referred to with like reference numerals and in which: [Figure 1] 1 illustrates an example of a networked computer system in which various embodiments may be implemented. [Figure 2] 1 illustrates example components of an audio management server computer according to disclosed embodiments. [Figure 3]1 shows an example of a neural network model for noise reduction. [Figure 4A] 1 shows an example of a feature extraction block. [Figure 4B] 10 shows another example of a feature extraction block. [Figure 5] 4 shows an example of a neural network model as a component of the neural model shown in FIG. 3. [Figure 6] 6 illustrates an example of a neural network model as a component of the neural network model illustrated in FIG. 5. [Figure 7] 4 shows an example of a neural network model as a component of the neural model shown in FIG. 3. [Figure 8] 1 illustrates an example of a process performed on an audio management server computer according to some embodiments described herein. [Figure 9] 1 is a block diagram illustrating a computer system upon which one embodiment of the present invention may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0007] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of example embodiments of the present invention. However, it will be apparent that example embodiments may be practiced without these specific details. Also, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the example embodiments.

[0008] Embodiments are described in the sections below according to the following outline: 1. Overall Overview 2. Example Computing Environment 3. Computer Component Examples 4. Functionality 4.1. Neural Network Model 4.1.1. Feature Extraction Block 4.1.2. U-NET Block Dense Blocks 4.1.2.1.1. Depthwise separable convolution using gating 4.1.2.2. Residual Blocks and Recurrent Layers Model Training 4.3. Model execution 5. Process Example 6. Hardware Implementation **

[0009] 1. Overall Overview A system and related method for suppressing noise and enhancing speech is disclosed. In some embodiments, the system trains a neural network model that takes banded energy corresponding to an original noisy waveform and generates speech values ​​indicating the amount of speech present in each band in each frame. These speech values ​​can be used to suppress noise by reducing the magnitude of frequencies in frequency bands where speech is unlikely to be present. The neural network model has low latency and can be used for real-time noise suppression. The neural model has a feature extraction block that performs some look-ahead. The feature extraction block is followed by an encoder with constant downsampling along the frequency domain, forming a contraction path. Convolutions along the contraction path are performed with increasingly larger dilation factors along the time dimension. The encoder is followed by a corresponding decoder with constant upsampling along the frequency domain, forming a dilation path. The decoder receives scaled output feature maps from the encoder at the corresponding level, and can therefore consider all features extracted from different receptive fields along the frequency dimension when determining how much speech is present in each frequency band in each frame.

[0010] In some embodiments, at run time, the system takes a noisy waveform and transforms it into the frequency domain, covering multiple perceptually motivating frequency bands in each frame. The system then runs a model to obtain a speech value for each frequency band in each frame. The system then applies the speech value to the original data in the frequency domain and transforms it back into an enhanced, noise-suppressed waveform.

[0011] The system has several technical advantages. It is designed to be low-latency yet accurate for real-time noise suppression. Low latency is achieved through a relatively small number of convolution kernels, such as eight two-dimensional kernels of size 1x1 or 3x3, in a lean convolutional neural network (CNN) model. Consolidation of the original frequency-domain data into perceptually stimulating bands further reduces the amount of computation. Depthwise separable convolutions, which tend to reduce execution time, are also applied where possible.

[0012] Accuracy is achieved through feature extraction for different receptive fields in the input data along the frequency dimension, which are used in combination to achieve dense classification. Certain feature extraction blocks that incorporate a look-ahead of a small number of frames, such as one or two frames, further contribute to feature richness. Dense blocks, in which the output feature maps of a convolutional layer are propagated to all subsequent convolutional layers, are also applied where possible. Furthermore, the neural model can be trained to predict not only the amount of speech present for each frequency band in each frame, but also the distribution of such amounts. Additional parameters of the distribution can be used to fine-tune the prediction.

[0013] 2. Example Computing Environment Figure 1 illustrates an example of a networked computer system in which various embodiments may be implemented. Figure 1 is shown in simplified schematic form for purposes of clarity of illustration, and other embodiments may include more, fewer, or different elements.

[0014] In some embodiments, the networked computer system includes an audio management server computer 102 (“server”), one or more sensors 104 or input devices, and one or more output devices 110, which are communicatively coupled through direct physical connections or via one or more networks 118.

[0015] In some embodiments, server 102 broadly represents one or more computers, virtual computing instances, and / or instances of applications programmed or configured with data structures and / or database records configured to host or perform functions related to low latency speech enhancement with noise reduction. Server 102 can comprise a server farm, a cloud computing platform, a parallel computer, or any other computing facility having sufficient computing power in data processing, data storage, and network communication for the functions described above.

[0016] In some embodiments, each of the one or more sensors 104 may include a microphone or other digital recording device that converts sound into an electrical signal. Each sensor is configured to transmit detected audio data to the server 102. Each sensor may include a processor or may be integrated into a typical client device, such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device.

[0017] In some embodiments, each of the one or more output devices 110 may include a speaker or other digital playback device that converts the electrical signal back into sound. Each output device is programmed to play audio data received from the server 102. Like the sensors, the output devices may include a processor or may be integrated into a typical client device, such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device.

[0018] The one or more networks 118 may be implemented by any medium or mechanism that provides for the exchange of data between the various elements of Figure 1. Examples of networks 118 include, but are not limited to, one or more of a cellular network communicatively coupled with a data connection to a computing device via a cellular antenna, a near field communication (NFC) network, a local area network (LAN), a wide area network (WAN), the Internet, a terrestrial or satellite link, etc.

[0019] In some embodiments, the server 102 is programmed to receive input audio data corresponding to sounds in a given environment from one or more sensors 104. The server 102 is then programmed to process the input audio data, which typically represents a mixture of speech and noise, to estimate how much speech is present in each frame of the input data. The server 102 is also programmed to update the input audio data based on the estimate to generate cleaned-up output audio data that is expected to contain less noise than the input audio data. The server 102 is further programmed to send the output audio data to one or more output devices.

[0020] 3. Computer Component Examples FIG. 2 illustrates example components of an audio management server computer according to disclosed embodiments. This diagram is for illustrative purposes only; the server 102 may have fewer or more functional or storage components. Each of the functional components may be implemented as a software component, a general-purpose or dedicated hardware component, a firmware component, or any combination thereof. Each of the functional components may also be coupled with one or more storage components (not shown). The storage components may be implemented using any of a relational database, an object database, a flat file system, or a JSON store. The storage components may be connected to the functional components locally or over a network using program calls, a remote procedure call (RPC) facility, or a messaging bus. Components may or may not be self-contained. Depending on implementation-specific or other considerations, components may be functionally or physically centralized or distributed.

[0021] In some embodiments, the server 102 includes a spectral transformation and banding block 204 , a model block 208 , an inverse banding block 212 , an input spectral multiplication block 218 , and an inverse spectral transformation block 222 .

[0022] In some embodiments, the server 102 receives a noisy waveform. At block 204, the server 102 segments the waveform into a sequence of frames through a spectral transformation, e.g., a 6-second long sequence with overlapping or non-overlapping 20 ms frames (resulting in 300 frames). The spectral transformation can be any of a variety of transforms, such as a short-time Fourier transform or a complex quadrature mirror filterbank (CQMF) transform, the latter of which tends to minimize aliasing artifacts. To ensure relatively high frequency resolution, the number of transform kernels / filters per 20 ms frame can be selected so that the frequency bin width is approximately 25 Hz.

[0023] In some embodiments, the server 102 then converts the sequence of frames into a vector of banded energy for, for example, 56 perceptually stimulating bands. Each perceptually stimulating band typically lies within a frequency domain, such as 120 Hz to 2,000 Hz, that corresponds to how the human ear processes sound. Therefore, capturing data within these perceptually stimulating bands means no loss of audio quality for the human ear. More specifically, the squared magnitudes of the output frequency bins of the spectral transformation are grouped into perceptually stimulating bands, with the number of frequency bins per band increasing at higher frequencies. The grouping strategy can be "soft," in which some spectral energy leaks across adjacent bands, or "hard," in which there is no leakage across bands.

[0024] In some embodiments, when the bin energies of a noisy frame are represented by x, a column vector of size p×1, where p represents the number of bins, the conversion to a vector of banded energies can be performed by calculating y=W*x, where y is a column vector of size q×1 representing the band energies of the noisy frame, W is a banding matrix of size q×p, and q represents the number of perceptually stimulated bands.

[0025] In some embodiments, the server 102 predicts a mask value indicating the amount of speech present for each band in each frame at block 208. At block 212, the server 102 converts the band mask values ​​back to spectral bin masks.

[0026] In some embodiments, when the band mask for y is represented by a column vector m_band of size q×1, the conversion to a bin mask can be performed by calculating m_bin=W_transpose*m_band, where m_bin is a column vector of size p×1 and W_transpose, of size p×q, is the transpose of W. At block 218, the server 102 multiplies the spectral bin mask by the spectral magnitude to achieve noise masking or reduction and obtain an estimated clean spectrum. Finally, at block 222, the server converts the estimated clean spectrum back into a waveform as an enhanced waveform (relative to the noise waveform) using any method known to those skilled in the art, such as an inverse transform (e.g., inverse CQMF), which can be communicated via an output device.

[0027] 4. Functionality 4.1. Neural Network Model 3 shows an example of a neural network model 300 for noise reduction, which represents one embodiment of block 208. In some embodiments, model 300 includes a block 308 for feature extraction and a block 340 based on a U-Net architecture, such as that described in arXiv:1505.04597v1 [cs.CV], published May 18, 2015, with some variations as described herein. The U-Net architecture has been shown to enable accurate localization of feature recognition and classification.

[0028] 4.1.1. Feature Extraction Block In some embodiments, at block 308 of FIG. 3, the server 102 extracts high-level features optimized for noise suppression tasks from the raw band energies. FIG. 4A shows an example of a feature extraction block that represents one embodiment of block 308. FIG. 4B shows another example of a feature extraction block. As shown in configuration 400A of FIG. 4A, for example, the server 102 can normalize the mean and variance of the band energies in a sequence of T frames (e.g., 56 of them) using a learnable batch normalization (BATCHNORM) layer 408 known to those skilled in the art. Alternatively, a global normalization can be pre-computed from a training set using techniques known to those skilled in the art.

[0029] In some embodiments, the server 102 can take future information into account when extracting the high-level features described above. For example, as shown in 400A of FIG. 4A, such a look-ahead can be implemented using a two-dimensional (2D), one-channel convolutional layer (conv2d) 406 with one or more kernels. The height of the kernel in the conv2d layer 406, corresponding to the number of n bands to evaluate each time, can be set to a small value, such as 3. The kernel size along the time axis depends on how much look-ahead is desired or tolerated. For example, without look-ahead, the kernel can cover the current frame and L past frames, such as two frames. When L future frames are allowed, the kernel size can be 2L+1, centered on the current frame, to match the 2L+1 frames in the input data for each time, such as 422 where L is 2 in 406. As shown in 400B of FIG. 4B, the look-ahead can also be implemented using a series of conv2d layers 410, 412, or more. In that case, each kernel has a small kernel size along the time axis. For example, L may be set to 1 for 410, 412, and all other similar layers. As a result, layer 410 can be matched with the original input data with 2L+1 look-ahead, e.g., 422 where L is 1 and leads to three kernels 428, and layer 412 can be matched with the output of layer 412. The server can gradually increase the receptive field in the input data using a series of conv2d layers as shown in FIG. 4B.

[0030] In some embodiments, the number of kernels in each conv2d layer can be determined based on the nature of the input audio stream, the amount of desired high-level features, the range of computing resource requirements, or other factors. For example, the number may be 8, 16, or 32. In addition, each of the conv2d layers in block 308 can be followed by a nonlinear activation function, such as a parametric rectified linear unit (PReLU), which can then be followed by a separate batch normalization layer to fine-tune the output of block 308.

[0031] In some embodiments, block 308 may be implemented using other signal processing techniques unrelated to artificial neural networks, such as those described in C. Kim and R.M. Stern, “Power-Normalized Cepstral Coefficients (PNCC) for Robust Speech Recognition,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 7, pp. 1315-1329, July 2016, doi: 10.1109 / TASLP.2016.2545928.

[0032] 4.1.2. U-NET Block In some embodiments, in block 340 of FIG. 3, the server 102 performs encoding of the feature data (to find more and better features) and then decoding to reconstruct the enhanced audio data before finally performing classification to determine how much speech is present. Thus, block 340 includes an encoder side on the left and a decoder side on the right, connected by block 350. The encoder includes one or more feature computation blocks, such as 310, 312, and 314, each followed by a frequency downsampler (DS), such as 316, 318, and 320, forming a contraction path. A dense block (DB), as described further below, is one implementation of such a feature computation block. Each triplet shown in the figure, e.g., (8, T, 64), contains the size of the input or output data of the feature computation block, with the first component representing the number of channels or feature maps, the second component representing a fixed number of frames along the time dimension, and the third component representing the size along the frequency dimension. These feature computation blocks capture higher-level features in a larger frequency context, as described further below. Block 350 includes a feature computation block for performing modeling covering all originally available perceptually stimulating bands. The decoder also includes one or more feature computation blocks, such as 320, 322, and 324, each followed by a frequency upsampler (US), such as 326, 328, and 330, forming a dilation path. These feature computation blocks in the dilation path, which rely on feature maps generated in the erosion path, combine to project discriminative features at multiple different levels, i.e., band-by-band level, in each frame, onto high-resolution space to obtain a dense classification, i.e., mask values. Due to this combination, the number of input channels (or feature maps) of each feature computation block in the dilation path can be twice the number of each feature computation block in the erosion path. However, the selection of the number of kernels in each computation block can determine the number of output channels, which becomes the number of input channels for the next feature computation block in the dilation path.

[0033] The server 102 generates a final mask value for each band in the frame via a classification block such as block 360 with a 1x1 2D kernel followed by a sigmoid nonlinear activation function (SIGMOID).

[0034] In some embodiments, for each frequency downsampler, the server 102 merges every two adjacent band energies using a conv2d layer with a kernel and a stride size of 2 along the frequency axis via normal or depthwise convolution. Alternatively, the conv2d layer may be replaced with a max-pooling layer. In either case, the width of the output feature map is halved after each frequency downsampler, thereby steadily expanding the receptive field within the input data. To enable such a continuous exponential reduction in the width of the output feature map, the server 102 pads the output of block 308 to a width that is a power of 2, which then becomes the input data to block 340. This padding can be done, for example, by adding zeros to both of the output feature maps of block 308.

[0035] In some embodiments, for each frequency upsampler, the server 102 uses a transposed conv2d layer corresponding to the conv2d layer at the same level in the encoder to restore the original number of band energies. The depth of block 340, i.e., the number of combinations of feature computation blocks and frequency downsamplers (and equivalently, the number of combinations of feature computation blocks and frequency upsamplers), may depend on the desired maximum receptive field, the amount of computational resources, or other factors.

[0036] In some embodiments, the server 102 connects the outputs of feature computation blocks in the encoder to the inputs of feature computation blocks in the decoder at the same level using skip connections, e.g., 342, 344, and 346, through which the decoder ultimately receives discriminative features of the input data at different levels for dense classification, as described above. For example, the feature map generated by block 310 is used as input data along with the feature map provided from frequency upsampler 330 to block 324 via skip connection 346. As a result, the number of channels of input data for each feature computation block in the decoder is twice the number of channels of input data for each dense block in the encoder.

[0037] In some embodiments, instead of simple concatenation, the server 102 learns a scalar multiplier for each skip connection, e.g., α1, α2, and α3, as shown in FIG. i contains N (e.g., 8) learnable parameters, which may be initialized to 1 at the beginning of training. Each of the learnable parameters is used to multiply the feature map produced by the corresponding feature computation block in the encoder to produce a scaled feature map, which is then concatenated with the feature map fed to the corresponding feature computation block in the decoder.

[0038] In some embodiments, server 102 can replace concatenation with addition. For example, the eight feature maps generated by block 310 can be added to the eight feature maps provided to dense block 324, with each of the eight additions performed component-wise. Such additions instead of concatenation reduce the number of feature maps used as input to each feature computation block in the decoder, reducing overall computation at the expense of some performance degradation.

[0039] Dense Blocks Figure 5 shows an example of a neural network model, which corresponds to one embodiment of block 310 and all other similar blocks in block 340 of Figure 3. The neural network model is based on a DenseNet structure, such as that described in arXiv:1608.06993v5 [cs.CV], published January 28, 2018, with some modifications as described herein. The DenseNet structure has been shown to mitigate the vanishing gradient problem, enhance feature propagation, promote feature reuse, and reduce the number of parameters.

[0040] In some embodiments, the server 102 uses block 500 as a feature computation block to further enhance feature propagation and dense classification. Block 500 outputs a feature map with N (e.g., 8) channels, the same as the number of feature map input data. Each channel also has the same time-frequency shape as the feature map of the input data. Block 500 has a series of convolutional layers, such as 520 and 530. The input data to each convolutional layer includes a concatenation of all output data of the preceding convolutional layer, thereby forming a dense connection. For example, the input data to layer 530 includes data 512, which may be the initial input data or output data from the previous convolutional layer, and data 522, which is the input data from layer 520.

[0041] In some embodiments, each convolutional layer includes a bottleneck layer with one or more 1×1 2D kernels, such as layer 504, to consolidate input data having K feature maps due to dense connections into a smaller number of feature maps. For example, each 1×1 2D kernel can be applied to each of the K / 2N feature maps in each group, effectively summing the K / 2N feature maps into one feature map to ultimately obtain 2N feature maps. Alternatively, a total of 2N 1×1 2D kernels can be applied to all feature maps to generate the 2D feature maps. Each 1×1 2D kernel can be followed by a nonlinear activation function, such as PReLU, and / or a batch normalization layer.

[0042] In some embodiments, each convolutional layer has a small conv2d layer with N kernels, followed by a bottleneck layer, to generate N feature maps, such as block 506 with a 3x3 conv2d layer. These small conv2d layers in successive convolutional layers in block 500 use exponentially increasing dilation along the time axis to model increasingly larger contextual information. For example, the dilation factor used in block 506 is 1, meaning no dilation for each kernel, and the dilation factor used in block 508 is 2, meaning the kernels are expanded by a factor of 2 along the time axis and the receptive field also increases in size by a factor of 2 in each dimension.

[0043] In some embodiments, between the convolutional layers of block 500, server 102 linearly projects the band energies into a learned space in a frequency mapping layer for a more integrated output, such as that described in arXiv:1904.11148v1 [cs.SD], April 25, 2019. Because the same kernel may produce different effects on the same audio data depending on which frequency band the audio data is located in, some integration of such effects across different bands is useful. For example, a frequency mapping layer 580 is placed midway through the depth of block 500.

[0044] In some embodiments, at the end of block 500, a layer 590 similar to the bottleneck layer with another 1x1 2D kernel may be used to generate an output tensor with N feature maps.

[0045] 4.1.2.1.1. Depthwise separable convolution using gating FIG. 6 illustrates an example of a neural network model that corresponds to an embodiment of block 506 and all other similar blocks shown in FIG. 5. In some embodiments, block 600 includes depthwise separable convolutions with nonlinear activation functions, such as a gated linear unit (GLU). As shown in FIG. 6, the first pass within GLU includes a depthwise small conv2d layer, such as a 3×3 conv2d layer 602, followed by a batch normalization (BATCHNORM) layer 604. The second pass within GLU similarly includes a 3×3 conv2d layer 606 followed by a batch normalization (BATCHNORM) layer 608, followed by a learnable gating function, such as a sigmoid nonlinear activation function (SIGMOID). As in the dense block shown in FIG. 5, the small conv2d layers in successive convolutional layers of block 500 use exponentially increasing dilation along the time axis to model increasingly larger contextual information. For example, blocks 602 and 606 in a convolutional layer corresponding to block 506 may be associated with a dilation factor of 1, while a similar block in the next convolutional layer, which may correspond to one embodiment of block 508, may be associated with a dilation factor of 2. The gating function identifies regions of the input data that are important for the task of interest. These two paths are combined by a Hadamard product operator 618. A 1×1 conv2d layer 612 learns the interconnections between the output feature maps produced by the combination of the two paths as part of a depthwise separable convolution. Layer 612 may be followed by a batch normalization layer 614 and a nonlinear activation function 616, such as PReLU.

[0046] 4.1.2.2. Residual Blocks and Recurrent Layers Figure 7 shows an example of a neural model that corresponds to an embodiment of block 310 and all other similar blocks shown in Figure 3. In some embodiments, block 500 shown in Figure 5, which also corresponds to an embodiment of block 310, can be replaced by a residual 700 block to reduce the number of connections. Block 700 has multiple convolutional layers, such as layers 720 and 730.

[0047] In some embodiments, each convolutional layer includes a bottleneck layer similar to block 504 shown in Figure 5, such as layer 704. The bottleneck layer may also be followed by a nonlinear activation, such as a PReLU, and / or a batch normalization layer.

[0048] In some embodiments, the convolutional layer also includes a small conv2d layer similar to block 506 shown in Figure 5, such as a 3x3 conv2d layer 706. This small conv2d block can be implemented using dilation, with a dilation factor that increases exponentially across successive convolutional layers. This small conv2d layer can be replaced by a depthwise separable convolution using gating, as shown in Figure 6.

[0049] In some embodiments, the convolutional layer has another 1x1 conv2d layer, such as layer 708, that matches the output of block 706 back to the input of block 704 in terms of size, specifically the number of channels or feature maps. The output is then added to the input data via a Hadamard product operator 710, which mitigates the vanishing gradient problem when training a network using backpropagation, since the gradient has a direct path from the output to the input without any multiplications between them. This conv1x1 layer may also be followed by a nonlinear activation layer, such as a PReLU, and / or a batch normalization layer.

[0050] In some embodiments, block 500 shown in FIG. 5, which also corresponds to one embodiment of block 310, may be replaced by a recurrent layer having at least one recurrent neural network (RNN). Using an RNN to model long time sequences can be an efficient approach. By "efficient," we mean that the RNN can model very long time sequences by maintaining an internal hidden state vector as a summary of all the history it has seen and generating an output for each new frame based on that vector. Compared to using dilation in the CNN layer, an RNN requires a much smaller buffer size for storing past information (only one vector versus 2d+1 vectors in a CNN, where d is the dilation factor).

[0051] Model Training In some embodiments, training of the neural network model 208 can be performed as an end-to-end process. Alternatively, the feature extraction block 308 and the U-Net block 340 can be trained separately, and the output of applying the feature extraction block 308 to real data can be used as training data for the U-Net block.

[0052] Diverse training data is used to train the neural network model 208 shown in Figure 2. In some embodiments, diversity incorporates speaker diversity by including natural speech with a wide range of speaking styles in the training data with respect to rate, emotion, and other attributes. Each training utterance may be speech from a single speaker or may represent a dialogue between multiple speakers.

[0053] In some embodiments, diversity results from including focused noise data, including reverberation data. A database such as AudioSet can be used as the seed noise database. The server 102 can filter each clip in the seed noise database using a class label that indicates a high probability that speech is present in the clip. For example, the class of "human voice" in a given ontology can be filtered. The seed noise database can be further filtered by applying any speech separation technique known to those skilled in the art to remove additional clips where speech is likely to be present. For example, any clip containing at least one frame (e.g., a 100 ms long frame) whose speech prediction has a root mean square energy above a threshold (e.g., 1e-3) can be removed.

[0054] In some embodiments, diversity is increased by including a wide range of intensity levels when mixing noise with speech. In creating the noisy signal, the server 102 can scale the clean speech signal and the noise signal to a predetermined maximum level, randomly adjust each down by one dB in a range, such as 0 to 30 dB, and randomly add the adjusted clean speech signal and adjusted noise signal together according to a predetermined minimum signal-to-noise ratio. Such a wide range of loudness levels has been found to help reduce over-suppression of speech (or under-suppression of noise).

[0055] In some embodiments, the diversity lies in the presence of data in different frequency bands. The server 102 may create a signal that has at least a certain percentage in a particular frequency band of a particular bandwidth, such as at least 20% in the 300 Hz to 500 Hz frequency band.

[0056] In some embodiments, the server 102 trains the neural network model 208 using any optimization process known to those skilled in the art, such as a stochastic gradient descent optimization algorithm in which weights are updated using a backpropagation algorithm. The neural network model 208 can minimize the mean squared error (MSE) loss between the predicted mask and the ground truth mask for each band in each frame. The ground truth mask can be calculated as the ratio of the speech energy to the sum of the speech energy and noise energy.

[0057] In some embodiments, because over-suppressing speech harms speech quality more than under-suppressing speech, the server 102 uses a weighted MSE that assigns a greater penalty to over-suppressing speech. Because the mask value generated by the neural network model 208 indicates the amount of speech present, when the predicted mask value is smaller than the ground truth mask value, less speech is predicted than the ground truth, and therefore more speech is being suppressed than necessary, leading to over-suppression of speech by the neural network model. For example, the weighted MSE may be calculated as follows:

number

number

number

[0058] In some embodiments, the neural network model 208 is trained to predict the distribution of audio across multiple different frequency bins within each band (rather than a single mask value). Specifically, the server 102 can train the model to predict the mean and variance of a Gaussian distribution for each band in each frame, with the mean representing the best prediction of the mask value by the neural network model 208. The loss function for a Gaussian distribution is:

number

number

[0059] In some embodiments, the variance prediction may be interpreted as a confidence in the mean prediction to reduce the occurrence of over-suppression of speech. If the mean prediction is relatively low, indicating a small amount of speech is present, and the variance prediction is relatively high, this may indicate a high probability of over-suppression of speech, and so the band mask may be scaled up. An example scaling function for generating an adjusted gain based on the standard deviation is:

number

[0060] In some embodiments, assuming a Gaussian distribution for each mask, the probability of each (target) mask value being observed is:

number

[0061] 4.3. Model execution In some embodiments, when look-ahead is implemented in the neural network model 208, specifically in the feature extraction block 308, the server 102 can accept individual frames or sets of frames as input data and generate at least a mask value for each frame as output data. For each convolutional layer with a kernel size greater than 1 along the time dimension, the server 102 maintains an internal buffer to store the history required to generate the output data. The buffer can be maintained as a queue with a size equal to the receptive field of the convolutional layer along the time dimension.

[0062] 5. Process Example Figure 8 illustrates an example of a process performed on an audio management server computer according to some embodiments described herein. Figure 8 is shown in simplified schematic form for purposes of clarity of illustration; other embodiments may include more, fewer, or different elements connected in various ways. Each of Figures 8 is intended to disclose an algorithm, plan, or outline that can be used to implement one or more computer programs or other software elements that, when executed, perform the functional improvements and technical advances described herein. Additionally, the flow diagrams herein are described at the same level of detail that those skilled in the art typically use to convey to one another the algorithms, plans, or specifications that form the basis of the software programs they plan to code or implement using their accumulated skill and knowledge.

[0063] In some embodiments, in step 802, the server 102 is programmed to receive input audio data covering multiple frequency bands along the frequency dimension in multiple frames along the time dimension, in some embodiments, the multiple frequency bands are perceptually stimulating bands, with higher frequencies covering more frequency bins.

[0064] In some embodiments, in step 804, the server 102 is programmed to train a neural network model having a feature extraction block that performs a look-ahead of a certain number of frames when extracting features from the input audio data, an encoder including a first series of blocks that generate feature maps corresponding to increasingly larger receptive fields in the input audio data along the frequency dimension, a decoder including a second series of blocks that receive as input feature maps the output feature maps generated by the encoder, and a classification block that generates, for each of a plurality of frequency bands in each frame of a plurality of frames, a speech value indicating the amount of speech present.

[0065] In some embodiments, the feature extraction block has a convolution kernel with a particular size along the time dimension, and the encoder and decoder do not have convolution kernels with a size along the time dimension greater than or equal to the particular size. In other embodiments, the feature extraction block, the first series of blocks, and the second series of blocks each generate a common number of feature maps.

[0066] In some embodiments, the feature extraction block comprises a batch normalization layer followed by a convolution layer with a two-dimensional convolution kernel.

[0067] In some embodiments, each block in the first series of blocks in the encoder comprises a feature computation block and a frequency downsampler, the feature computation block comprising a series of convolutional layers.

[0068] In some embodiments, the output data of a convolutional layer in the series of convolutional layers is fed to all subsequent convolutional layers in the series. The series of convolutional layers implements progressively larger dilation along the time dimension. In other embodiments, each of the series of convolutional layers has a depth-separable convolution block with a gating mechanism.

[0069] In some embodiments, each of the series of convolutional layers has a residual block with a series of convolutional blocks, including a first convolutional block with a first 1×1 two-dimensional convolutional kernel and a last convolutional block with a last 1×1 two-dimensional convolutional kernel.

[0070] In some embodiments, output data of a feature computation block in a block of the first series of blocks is scaled by a learnable weight to form scaled output data, and the scaled output data is communicated to a block of the second series of blocks in the decoder via a skip connection.

[0071] In some embodiments, the frequency downsampler of a block in the first series of blocks has a convolution kernel with a stride size greater than 1 along the frequency dimension.

[0072] In some embodiments, each block in the second series of blocks includes a feature calculation block and a frequency upsampler, and the feature calculation block in a block in the second series of blocks receives first output data from the feature calculation block in a block in the first series of blocks and second output data from the frequency upsampler of a preceding block in the second series of blocks, and the first output data and the second output data are concatenated or added together to form specific input data for the feature calculation block in the block in the second series of blocks.

[0073] In some embodiments, the classification block has a 1x1 two-dimensional convolution kernel and a non-linear activation function.

[0074] In some embodiments, the neural network model further comprises a feature calculation block, which is the output data of the encoder and the input data of the decoder.

[0075] In some embodiments, the server 102 is programmed to perform training using a loss function between the predicted speech value and the ground truth speech value for each frequency band of the plurality of frequency bands in each frame, increasing the weight in the loss function if the predicted speech value corresponds to over-suppression of speech and decreasing the weight in the loss function if the predicted speech value corresponds to under-suppression of speech. In some embodiments, the classification block further generates a distribution of the amount of speech across a frequency band of the plurality of frequency bands in the frame, and the speech value is the mean of the distribution.

[0076] In some embodiments, the input audio data comprises data corresponding to speech of different speeds or emotions, data containing different levels of noise, or data corresponding to different frequency bins.

[0077] In some embodiments, in step 806, the server 102 is programmed to receive new audio data having one or more frames.

[0078] In some embodiments, in step 808, the server 102 is programmed to run a neural network model on the new audio data to generate new audio values ​​for each frequency band of the plurality of frequency bands in each frame of the one or more frames.

[0079] In some embodiments, at step 810, the server 102 is programmed to generate new output data that suppresses noise in the new audio data based on the new voice values.

[0080] In some embodiments, in step 812, the server 102 is programmed to send the new output data.

[0081] In some embodiments, the server 102 is programmed to receive an input waveform. The server 102 is then programmed to convert the input waveform into raw audio data covering multiple frequency bins along the frequency dimension in one or more frames along the time dimension. The server 102 is then programmed to convert the raw audio data into new audio data by grouping the multiple frequency bins into multiple frequency bands. The server 102 is then programmed to perform inverse banding on the new audio values ​​to generate updated audio values ​​for each frequency bin of the multiple frequency bins in each frame of the one or more frames. The server 102 is further programmed to then apply the updated audio values ​​to the raw audio data to generate new output data. Finally, the server 102 is programmed to convert the new output data into an enhanced waveform.

[0082] 6. Hardware Implementation According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and / or other computing devices coupled using a network, such as a packet data network. The computing device may be hardwired to perform the techniques, may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) permanently programmed to perform the techniques, or may include at least one general-purpose hardware processor programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such a computing device may also combine custom hardwired logic, ASIC, or FPGA with custom programming to achieve the described techniques. The computing device may be a server computer, a workstation, a personal computer, a portable computer system, a handheld device, a mobile computing device, a wearable device, a body-worn or implantable device, a smartphone, a smart appliance, an internetworking device, an autonomous or semi-autonomous device such as a robot or an unmanned ground vehicle or aircraft, any other electronic device incorporating hardwired and / or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0083] Figure 9 is a block diagram illustrating an example computer system in which one embodiment may be implemented. In the example of Figure 9, computer system 900 and instructions for implementing the disclosed techniques in hardware, software, or a combination of hardware and software are represented schematically, for example, as boxes and circles, at the same level of detail commonly used by those skilled in the art to which this disclosure pertains to communicate about computer architectures and computer system implementations.

[0084] Computer system 900 includes an input / output (I / O) subsystem 902, which may include buses and / or other communication mechanism(s) for communicating information and / or instructions between components of computer system 900 via electronic signal paths. I / O subsystem 902 may include an I / O controller, a memory controller, and at least one I / O port. Electronic signal paths are represented schematically in the figures, for example, as lines, single-headed arrows, or double-headed arrows.

[0085] At least one hardware processor 904 is coupled to the I / O subsystem 902 for processing information and instructions. The hardware processor 904 may include, for example, a general-purpose microprocessor or microcontroller and / or a special-purpose microprocessor, such as for embedded systems, or a graphics processing unit (GPU), or a digital signal processor, or an ARM processor. The processor 904 may have an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

[0086] Computer system 900 includes one or more units of memory 906, such as main memory, coupled to I / O subsystem 902 for electronically and digitally storing instructions and data executed by processor 904. Memory 906 may include volatile memory, such as various forms of random access memory (RAM) or other dynamic storage devices. Memory 906 may also be used to store temporary variables or other intermediate information during execution of instructions executed by processor 904. Such instructions, when stored on a non-transitory computer-readable storage medium accessible to processor 904, may turn computer system 900 into a special-purpose machine customized to perform the operations specified in the instructions.

[0087] Computer system 900 further includes non-volatile memory, such as read-only memory (ROM) 908 or other static storage device, coupled to I / O subsystem 902 for storing information and instructions for processor 904. ROM 908 may include various forms of programmable ROM (PROM), such as erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM). A unit of persistent storage 910, which may include various forms of non-volatile random access memory (NVRAM), such as flash memory, or solid-state storage, magnetic disks, or optical disks, such as CD-ROMs or DVD-ROMs, may be coupled to I / O subsystem 902 for storing information and instructions. Storage 910 is an example of a non-transitory computer-readable medium that may be used to store instructions and data that, when executed by processor 904, cause computer-implemented methods to execute the techniques herein.

[0088] The instructions in memory 906, ROM 908, or storage 910 may comprise one or more instruction sets organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps. The instructions may include operating system and / or system software; one or more libraries supporting multimedia, programming, or other functionality; data protocol instructions or stacks implementing TCP / IP, HTTP, or other communications protocols; file processing instructions for interpreting and rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a graphical user interface (GUI), command line interface, or text user interface; application software such as an office suite, Internet access application, design and manufacturing application, graphics application, audio application, software engineering application, educational application, game, or other application. The instructions may implement a web server, a web application server, or a web client. The instructions may be organized as a presentation layer, an application layer, and a data storage layer, such as a relational database system using Structured Query Language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0089] The computer system 900 may be coupled to at least one output device 912 via the I / O subsystem 902. In one embodiment, the output device 912 is a digital computer display. Examples of displays that may be used in various embodiments include a touchscreen display, a light-emitting diode (LED) display, a liquid crystal display (LCD), or an electronic paper display. The computer system 900 may include other types of output device(s) 912 instead of, or in addition to, a display device. Examples of other output devices 912 include a printer, a ticket printer, a plotter, a projector, a sound or video card, a speaker, a buzzer or piezoelectric device or other audible device, a lamp or LED or LCD indicator, a tactile device, an actuator, or a servo.

[0090] At least one input device 914 is coupled to the I / O subsystem 902 for communicating signals, data, command selections, or gestures to the processor 904. Examples of input device 914 include touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and / or various types of sensors, such as force sensors, motion sensors, thermal sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors, and / or various types of transceivers, such as wireless, radio frequency (RF), or infrared (IR) transceivers, such as cellular or Wi-Fi, and global positioning system (GPS) transceivers.

[0091] Another type of input device is a control device 916, which may perform cursor control or other automated control functions, such as navigation in a graphical interface on a display screen, instead of or in addition to input functions. The control device 916 may be a touchpad, mouse, trackball, or cursor direction keys for communicating directional information and command selections to the processor 904 and for controlling cursor movement on the display 912. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to define a position in a plane. Other types of input devices are wired, wireless, or optical control devices, such as a joystick, wand, console, steering wheel, pedals, gear shift mechanism, or other type of control device. The input device 914 may include a combination of multiple different input devices, such as a video camera and a depth sensor.

[0092] In another embodiment, computer system 900 may include an Internet of Things (IoT) device in which one or more of output device(s) 912, input device(s) 914, and control device(s) 916 are omitted. Alternatively, in such an embodiment, input device(s) 914 may include one or more cameras, motion detectors, thermometers, microphones, earthquake detectors, other sensors or detectors, measuring devices, or encoders, and output device(s) 912 may include a dedicated display, such as a single-wire LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator, or a servo.

[0093] If computer system 900 is a mobile computing device, input device 914 may have a Global Positioning System (GPS) receiver coupled to a GPS module that can triangulate against multiple GPS satellites to determine and generate geolocation or position data, such as latitude-longitude values, for the geophysical location of computer system 900. Output device 912 may include hardware, software, firmware, and interfaces for generating position report packets, notifications, pulse or heartbeat signals, or other recurring data transmissions defining the location of computer system 900, alone or in combination with other application-specific data, to host 924 or server 930.

[0094] Computer system 900 may implement the techniques described herein using customized hardwired logic, at least one ASIC or FPGA, firmware, and / or program instructions or logic that, when loaded and used or executed in conjunction with the computer system, cause or program the computer system to operate as a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 900 in response to processor 904 executing at least one sequence of at least one instruction contained in main memory 906. Such instructions may be read into main memory 906 from another storage medium, such as storage 910. Execution of the sequences of instructions contained in main memory 906 causes processor 904 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0095] The term "storage medium," as used herein, refers to any non-transitory medium that stores instructions and / or data that cause a machine to operate in a specific fashion. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage 910. Volatile media include, for example, dynamic memory, such as memory 906. Common forms of storage media include, for example, hard disks, solid-state drives, flash drives, magnetic data storage media, any optical or physical data storage media, memory chips, or the like.

[0096] Storage media are distinct from, but may be used in conjunction with, transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a bus in I / O subsystem 902. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0097] Various forms of media may be involved in carrying at least one sequence of at least one instructions to processor 904 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions over a communications link, such as a fiber optic or coaxial cable or a telephone line, using a modem. A modem or router local to computer system 900 may receive the data over the communications link and convert the data so that it can be read by computer system 900. For example, a receiver such as a radio frequency antenna or infrared detector may receive the data carried in a radio or optical signal and appropriate circuitry can provide the data to I / O subsystem 902, such as placing the data on a bus. I / O subsystem 902 carries the data to memory 906, from which processor 904 retrieves and executes the instructions. The instructions received by memory 906 may optionally be stored on storage 910 either before or after execution by processor 904.

[0098] Computer system 900 also includes a communication interface 918 coupled to bus 902. Communication interface 918 provides a two-way data communication coupling to network link(s) 920 that is directly or indirectly connected to at least one communications network, such as, for example, a network 922 or a public or private cloud on the Internet. For example, communication interface 918 may be an Ethernet networking interface, an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing a data communication connection to a corresponding type of communications line, such as, for example, an Ethernet cable or any type of metallic or fiber optic line or a telephone line. Network 922 broadly represents a local area network (LAN), a wide area network (WAN), a campus network, an internetwork, or any combination thereof. Communication interface 918 may comprise a LAN card providing a data communication connection to a compatible LAN, or a wired cellular radiotelephone interface for transmitting or receiving cellular data in accordance with a cellular radiotelephone wireless networking standard, or a wired satellite radio interface for transmitting or receiving digital data in accordance with a satellite wireless networking standard. In any such implementation, communication interface 918 sends and receives electrical, electromagnetic or optical signals over signal paths that carry digital data streams representing various types of information.

[0099] Network link 920 typically provides electrical, electromagnetic, or optical data communication to other data devices directly or through at least one network, for example using satellite, cellular, Wi-Fi, or BLUETOOTH technology. For example, network link 920 may provide a connection through network 922 to a host computer 924.

[0100] Further, network link 920 may provide connectivity through network 922 or to other computing devices through internetworking equipment and / or computers operated by an Internet Service Provider (ISP) 926. ISP 926 provides data communication services through a worldwide packet data communication network represented as Internet 928. A server computer 930 may be coupled to Internet 928. Server 930 broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or a computer running a containerized program system such as DOCKER or KUBERNETES. Server 930 may represent an electronic digital service implemented using two or more computers or instances and accessed and used by sending web service requests, uniform resource locator (URL) strings with parameters in HTTP payloads, API calls, app service calls, or other service calls. Computer system 900 and server 930 may form elements of a distributed computing system that includes other computers, processing clusters, server farms, or other organizations of computers that cooperate to perform tasks or run applications or services. The server 930 may include one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps.The instructions may include operating system and / or system software; one or more libraries supporting multimedia, programming, or other functionality; data protocol instructions or stacks implementing TCP / IP, HTTP, or other communications protocols; file formatting instructions for interpreting or rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a graphical user interface (GUI), a command line interface, or a text user interface; and application software such as an office suite, Internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or other applications. Server 930 may include a web application server that hosts a presentation layer, an application layer, and a data storage layer, such as a relational database system using Structured Query Language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0101] Computer system 900 can send messages and receive instructions and data, including program code, through the network(s), network link 920 and communication interface 918. In the Internet example, a server 930 might transmit a requested code for an application program through Internet 928, ISP 926, local network 922 and communication interface 918. The received code may be executed by processor 904 as it is received, and / or stored in storage 910, or other non-volatile storage for later execution.

[0102] The execution of instructions described in this section may implement a process in the form of a running computer program instance, which consists of program code and its current activity. Depending on the operating system (OS), a process may consist of multiple threads of execution that execute instructions simultaneously. In this context, a computer program is a passive collection of instructions, while a process is the actual execution of those instructions. Several processes may be associated with the same program; for example, opening several instances of the same program often means that more than one process is running. Multitasking may be implemented to allow multiple processes to share the processor 904. Although each processor 904 or each core of a processor executes one task at a time, computer system 900 may be programmed to implement multitasking, allowing each processor to switch between multiple executing tasks without having to wait for each task to finish. In one embodiment, switching may be performed when a task performs an input / output operation, when a task indicates that it is available to switch, or upon a hardware interrupt. Time sharing may be implemented to enable fast response for interactive user applications by quickly performing context switching to provide the appearance of concurrent execution of multiple processes. In one embodiment, for security and reliability, the operating system may prevent direct communication between independent processes and provide a strictly mediated and controlled inter-process communication facility.

[0103] 7. Extensions and Substitutions In the foregoing specification, embodiments of the present disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the specification and drawings should be viewed in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the present disclosure, and what is intended by the applicant to be the scope of the present disclosure, is the literal scope and equivalent scope of the set of claims issued from this application, including any subsequent amendments in the specific form in which such claims arise.

Claims

1. 1. A computer-implemented method for suppressing noise and enhancing speech, comprising: receiving, by a processor, input audio data having one or more frames; executing, by the processor, a neural network model using the input audio data to generate an audio value for each frequency band of a plurality of frequency bands in each frame of the one or more frames, the neural network model comprising: a feature extraction block; An encoder; A decoder; a classification block; generating output audio data based on the audio values, the output audio data suppressing noise in the input audio data; 1. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the feature extraction block is configured to perform a look-ahead of a certain number of frames to extract features from the input audio data.

3. 2. The computer-implemented method of claim 1, wherein the encoder comprises a first series of blocks that generate first feature maps that correspond to increasingly larger receptive fields within the input audio data along a frequency dimension.

4. 4. The computer-implemented method of claim 3, wherein each block in the first series of blocks comprises a feature computation block and a frequency downsampler, the feature computation block comprising a series of convolutional layers, wherein output data of a convolutional layer in the series is fed to all subsequent convolutional layers in the series, and the series of convolutional layers implements increasingly larger dilation along the time dimension.

5. 2. The computer-implemented method of claim 1, wherein the decoder comprises a second series of blocks that receives as input feature maps the output feature maps produced by the encoder and produces second feature maps.

6. 6. The computer-implemented method of claim 5, wherein the classification block is configured to receive the second feature map and generate, for each frequency band of the plurality of frequency bands in each frame of the one or more frames, a speech value indicative of an amount of speech present.

7. receiving an input waveform; converting the input waveform into raw audio data covering a plurality of frequency bins along the frequency dimension in the one or more frames along the time dimension; converting the raw audio data to the input audio data by grouping the plurality of frequency bins into the plurality of frequency bands; performing inverse banding on the audio values ​​to generate updated audio values ​​for each frequency bin of the plurality of frequency bins in each frame of the one or more frames; applying the updated speech values ​​to the raw audio data to generate new output data; converting the new output data into an enhanced waveform; The computer-implemented method of claim 1 further comprising:

8. The computer-implemented method of claim 1 , wherein the plurality of frequency bands comprises perceptually stimulating bands that are higher in frequency and cover more frequency bins.

9. 3. The computer-implemented method of claim 1, wherein the feature extraction block has a convolution kernel with a particular size along the time dimension, the particular size being larger than the size along the time dimension of any convolution kernel in the encoder or the decoder.

10. 3. The computer-implemented method of claim 1 or 2, wherein the feature extraction block comprises a batch normalization layer followed by a convolution layer with a two-dimensional convolution kernel.

11. 5. The computer-implemented method of claim 4, wherein each of the series of convolutional layers has a depth-separable convolutional block with a gating mechanism.

12. 5. The computer-implemented method of claim 4, wherein each of the series of convolutional layers has a residual block having a series of convolutional blocks including a first convolutional block having a first 1×1 two-dimensional convolutional kernel and a last convolutional block having a last 1×1 two-dimensional convolutional kernel.

13. 4. The computer-implemented method of claim 3, wherein a frequency downsampler for a block in the first series of blocks has a convolution kernel with a stride size greater than 1 along the frequency dimension.

14. The computer-implemented method of claim 5 , wherein each block in the second series of blocks comprises a feature calculation block and a frequency upsampler.

15. The computer-implemented method of claim 1 , wherein the classification block has a 1×1 two-dimensional convolution kernel and a nonlinear activation function.

16. 2. The computer-implemented method of claim 1, wherein the classification block is configured to generate a distribution of the amount of speech across a frequency band of the plurality of frequency bands in a frame, and the corresponding speech value of the frame is the mean of the distribution.

17. The computer-implemented method of claim 1 , wherein the input audio data comprises data corresponding to speech of different speeds or emotions, data containing different levels of noise, or data corresponding to different frequency bins.

18. The computer-implemented method of claim 1 , wherein the neural network model further comprises a feature calculation block that is the output data of the encoder and the input data of the decoder.

19. memory, and one or more processors coupled to the memory; receiving input audio data having one or more frames; executing a neural network model using the input audio data to generate an audio value for each frequency band of a plurality of frequency bands in each frame of the one or more frames, the neural network model a feature extraction block; An encoder; A decoder; a classification block; generating output audio data based on the audio values, the output audio data suppressing noise in the input audio data; one or more processors configured to perform A computer system having:

Images