Speech enhancement based on deep learning

Abstract

The invention relates to speech enhancement based on deep learning. A system and related methods for noise suppression and speech enhancement are disclosed. The system trains a neural network model that obtains a band energy corresponding to an original band noise waveform and generates a speech value indicating an amount of speech present in each frequency band at each frame. The neural model includes a feature extraction block that implements a certain look-ahead. The feature extraction block is followed by an encoder that performs stable downsampling along the frequency domain, thereby forming a contracted path. The encoder is followed by a corresponding decoder that performs stable up-sampling along the frequency domain, thereby forming an expanded path. The decoder receives a scaled output feature map from an encoder of a corresponding level. The decoder is followed by a classification block that generates a speech value indicating an amount of speech present for each of a plurality of frequency bands at each of a plurality of frames.

Description

[0001] This application is a divisional application of the invention patent application with an international application date of October 29, 2021, a national application number of 202180073792.3 (international application number PCT / US2021 / 057378), and an invention name of “Speech Enhancement Based on Deep Learning”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Application No. 63 / 115,213 filed on November 18, 2020, U.S. Provisional Application No. 63 / 221,629 filed on July 14, 2021, and International Patent Application No. PCT / CN2020 / 124635 filed on October 29, 2020, all of which are incorporated herein by reference in their entirety. Technical Field

[0004] The present application relates to speech denoising. More specifically, the example embodiment(s) described below relate to applying deep learning models to generate frame-based inferences from large speech contexts. Background Art

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0006] It is often difficult to accurately remove noise from a mixed signal of speech and noise, given that there may be different forms of speech and different types of noise. Suppressing noise in real time can be particularly challenging. Summary of the invention

[0007] A system and related method for suppressing noise and enhancing speech are disclosed. The method includes: receiving input audio data by a processor, the input audio data covering multiple frequency bands along a frequency dimension at multiple frames along a time dimension; training a neural network model by the processor, the neural network model including: a feature extraction block, the feature extraction block implementing a specific number of frames of look-ahead when extracting features from the input audio data; an encoder, the encoder including a first series of blocks that generate a first feature map, the first feature map corresponding to increasingly larger receptive fields in the input audio data along the frequency dimension; a decoder, the decoder including a second series of blocks, the second series of blocks receiving the output feature map generated by the encoder as an input feature map and generating a second feature map; and a classification block, the classification block receiving the second feature map and generating a speech value, the speech value indicating the amount of speech present for each of the multiple frequency bands at each of the multiple frames; receiving new audio data including one or more frames; executing the neural network model on the new audio data to generate a new speech value for each of the multiple frequency bands at each of the one or more frames; generating new output data suppressing noise in the new audio data based on the new speech value; transmitting the new output data. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Example embodiment(s) of the present invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0009] Figure 1 An example networked computer system is illustrated in which various embodiments may be practiced.

[0010] Figure 2 Illustrated are example components of an audio management server computer in accordance with the disclosed embodiments.

[0011] Figure 3 Illustration of an example neural network model for denoising.

[0012] Figure 4A An example feature extraction block is illustrated.

[0013] Figure 4B Another example feature extraction block is illustrated.

[0014] Figure 5 The diagram shows Figure 3 An example neural network model with components of a neural model illustrated in FIG.

[0015] Figure 6 The diagram shows Figure 5 An example neural network model with components of a neural network model illustrated in FIG.

[0016] Figure 7 The diagram shows Figure 3 An example neural network model with components of a neural model illustrated in FIG.

[0017] Figure 8 An example process performed by an audio management server computer according to some embodiments described herein is illustrated.

[0018] Fig. 9 is a block diagram illustrating a computer system upon which embodiments of the invention may be implemented. DETAILED DESCRIPTION

[0019] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiment(s) of the present invention. However, it is apparent that the exemplary embodiment(s) may be practiced without these specific details. In other instances, known structures and devices are shown in block diagram form in order to avoid unnecessary ambiguity of the exemplary embodiment(s).

[0020] The embodiments are described in the following subsections according to the following summary:

[0021] 1. General Overview

[0022] 2. Sample computing environment

[0023] 3. Sample Computer Parts

[0024] 4. Functional Description

[0025] Neural network model

[0026] 4.1.1. Feature extraction block

[0027] 4.1.2.U-NET block

[0028] 4.1.2.1. Dense Blocks

[0029] 4.1.2.1.1. Depthwise Separable Convolution with Gating

[0030] 4.1.2.2. Residual Blocks and Recursive Layers

[0031] Model Training

[0032] Model Execution

[0033] 5. Example Process

[0034] 6. Hardware Implementation

[0035] **

[0036] 1. General Overview

[0037] A system and related method for suppressing noise and enhancing speech are disclosed. In some embodiments, the system trains a neural network model that takes band energies corresponding to an original noisy waveform and produces speech values ​​that indicate the amount of speech present in each frequency band at each frame. These speech values ​​can be used to suppress noise by reducing the frequency amplitudes in those frequency bands where speech is less likely to be present. The neural network model has low latency and can be used for real-time noise suppression. The neural model includes a feature extraction block that implements some kind of look-ahead. The feature extraction block is followed by an encoder that performs stable downsampling along the frequency domain, thereby forming a contraction path. The convolution along the contraction path is performed with increasingly larger dilation factors along the time dimension. The encoder is followed by a corresponding decoder that performs stable upsampling along the frequency domain, thereby forming an expansion path. The decoder receives a scaled output feature map from the encoder at the corresponding level so that features extracted from different receptive fields along the frequency dimension can be taken into account when determining how much speech is present in each frequency band at each frame.

[0038] In some embodiments, at runtime, the system acquires a noisy waveform, converts it into a frequency domain covering multiple perceptually excited frequency bands at each frame. The system then executes the model to obtain speech values ​​for each frequency band at each frame. The system then applies the speech values ​​to the raw data in the frequency domain and converts it back into an enhanced noise suppressed waveform.

[0039] The system has various technical advantages. The system is designed to be accurate while having low latency for real-time noise suppression. In a lean convolutional neural network (CNN) model, low latency is achieved via a relatively small number of relatively small convolution kernels (e.g., eight two-dimensional kernels of size 1 by 1 or 3 by 3). Merging the initial frequency domain data into perceptually motivated bands further reduces the amount of computation. Where possible, depth-separable convolutions are also applied, which tend to reduce execution time.

[0040] Accuracy is achieved by feature extraction of different receptive fields along the frequency dimension in the input data, which are used in combination to achieve dense classification. Specific feature extraction blocks that incorporate a look-ahead of a small number of frames (such as one or two frames) further promote feature richness. Where possible, the output feature maps of the convolutional layer are also applied to the dense blocks of all subsequent convolutional layers. In addition, the neural model can be trained to predict not only the amount of speech in each frequency band at each frame, but also the distribution of such amounts. Additional parameters of the distribution can be used to fine-tune the prediction.

[0041] 2. Sample computing environment

[0042] Figure 1An example networked computer system is illustrated in which various embodiments may be practiced. Figure 1 The examples are shown in a simplified schematic format for purposes of illustrating clarity, and other embodiments may include more, fewer, or different elements.

[0043] In some embodiments, the networked computer system includes an audio management server computer 102 (“server”), one or more sensors 104 or input devices, and one or more output devices 110 , which are communicatively coupled by direct physical connections or via one or more networks 118 .

[0044] In some embodiments, server 102 broadly represents one or more computers, virtual computing instances, and / or instances of applications that are programmed or configured with data structures and / or database records that are arranged to host or perform functions related to low-latency speech enhancement through noise reduction. Server 102 may include a server farm, a cloud computing platform, a parallel computer, or any other computing facility with sufficient computing power in terms of data processing, data storage, and network communications for the above functions.

[0045] In some embodiments, each of the one or more sensors 104 may include a microphone or another digital recording device that converts sound into electrical signals. Each sensor is configured to transmit detected audio data to the server 102. Each sensor may include a processor or may be integrated into a typical client device, such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device.

[0046] In some embodiments, each of the one or more output devices 110 may include a speaker or another digital playback device that converts electrical signals back into sound. Each output device is programmed to play audio data received from the server 102. Similar to sensors, output devices may include processors or may be integrated into typical client devices such as desktop computers, laptop computers, tablet computers, smartphones, or wearable devices.

[0047] One or more networks 118 may be provided by Figure 1 Examples of network 118 include, but are not limited to, one or more cellular networks (communicatively coupled via a cellular antenna with a data connection to a computing device), a near field communication (NFC) network, a local area network (LAN), a wide area network (WAN), the Internet, terrestrial or satellite links, etc.

[0048] In some embodiments, the server 102 is programmed to receive input audio data corresponding to sounds in a given environment from one or more sensors 104. The server 102 is programmed to then process the input audio data, which is typically a mixture of speech and noise, to estimate how much speech is present in each frame of the input data. The server 102 is also programmed to update the input audio data based on the estimated value to generate cleaned output audio data that is expected to contain less noise than the input audio data. In addition, the server 102 is programmed to send the output audio data to one or more output devices.

[0049] 3. Sample Computer Parts

[0050] Figure 2 Example components of an audio management server computer according to the disclosed embodiment are illustrated. The figure is for illustrative purposes only, and the server 102 may include fewer or more functional components or storage components. Each functional component may be implemented as a software component, a general or special hardware component, a firmware component, or any combination thereof. Each functional component may also be coupled to one or more storage components (not shown). The storage component may be implemented using any of a relational database, an object database, a flat file system, or a JSON storage. The storage component may be connected to the functional component locally or through a network using programmatic calls, a remote procedure call (RPC) facility, or a message bus. The components may or may not be independent. Depending on implementation-specific or other considerations, the components may be functionally or physically centralized or distributed.

[0051] In some embodiments, the server 102 includes a spectral transform and labeling block 204 , a model block 208 , an inverse striping block 212 , an input spectrum multiplication block 218 , and an inverse spectral transform block 222 .

[0052] In some embodiments, the server 102 receives a noisy waveform. In block 204, the server 102 segments the waveform into a sequence of frames, such as a six-second sequence with or without overlap, with 20 ms frames (yielding 300 frames), by spectral transforming. The spectral transform can be any of a variety of transforms, such as a short-time Fourier transform or a complex quadrature mirror filter bank (CQMF) transform, the latter of which tends to produce minimal aliasing artifacts. To ensure relatively high frequency resolution, the number of transform kernels / filters per 20 ms frame can be selected so that the frequency bin width is approximately 25 Hz.

[0053] In some embodiments, the server 102 then converts the frame sequence into band energy vectors of, for example, 56 perceptually excited frequency bands. Each perceptually excited frequency band is typically located in a frequency domain such as 120 Hz to 2,000 Hz, which matches the way the human ear processes speech, so that capturing data in these perceptually excited frequency bands means that there is no loss of speech quality to the human ear. More specifically, the squared amplitudes of the output frequency bins of the spectral transform are grouped into perceptually excited frequency bands, where the number of frequency bins per band increases at higher frequencies. The grouping strategy can be "soft" in the case of some spectral energy leakage across adjacent frequency bands, or "hard" in the case of no leakage across frequency bands.

[0054] In some embodiments, when the bin energy of a noisy frame is represented by a column vector x of size p times 1 ( in , p represents the number of bins), the conversion to a band energy vector can be performed by calculating y=W*x, where y is a column vector of size q times 1 representing the band energy of the noisy frame, W is a striping matrix of size q times p, and q represents the number of perceptually excited frequency bands.

[0055] In some embodiments, in block 208, the server 102 predicts a mask value for each frequency band at each frame, the mask value indicating the amount of speech present. In block 212, the server 102 converts the frequency band mask values ​​back to spectral bin masks.

[0056] In some embodiments, when the band mask of y is represented by a column vector m_band of size q times 1, the conversion to the bin mask can be performed by calculating m_bin=W_transpose*m_band, where m_bin is a column vector of size p times 1 and W_transpose of size p times q is the transpose of W. In block 218, the server 102 multiplies the spectrum amplitude mask by the spectrum amplitude to achieve masking or reduction of noise and obtain an estimated clean spectrum. Finally, in block 222, the server converts the estimated clean spectrum back into a waveform as an enhanced waveform (relative to the noise waveform) using any method known to those skilled in the art, such as an inverse transform (such as inverse CQMF), which can be transmitted via an output device.

[0057] 4. Functional Description

[0058] Neural network model

[0059] Figure 3An example neural network model 300 for denoising is illustrated that represents an embodiment of block 208. In some embodiments, model 300 includes block 308 for feature extraction and block 340 based on a U-Net structure, such as the U-Net structure described in arXiv:1505.04597v1[cs.CV] May 18, 2015, but with some variations, as described herein. The U-Net structure has been shown to be capable of accurate feature recognition and classification.

[0060] 4.1.1. Feature extraction block

[0061] In some embodiments, Figure 3 In block 308 , the server 102 extracts high-level features optimized for the noise suppression task from the raw band energies. Figure 4A An example feature extraction block representing an embodiment of block 308 is illustrated. Figure 4B Another example feature extraction block is shown. Figure 4A As illustrated in the structure 400A in FIG. 1 , the server 102 may normalize the mean and variance of the band energies (e.g., 56 of them) in the sequence of T frames using a learnable batch normalization layer 408 known to those skilled in the art. Alternatively, a global normalization may be pre-computed from a training set using techniques known to those skilled in the art.

[0062] In some embodiments, server 102 may consider future information when extracting the above high-level features. Figure 4A As illustrated in 400A in , such lookahead can be implemented with a two-dimensional (2D) single-channel convolutional layer (conv2d) layer 406 having one or more kernels. The height of the kernel in conv2d layer 406 corresponding to the number of frequency bands to be evaluated at a time can be set to a small value, such as three. The size of the kernel along the time axis depends on how much lookahead is desired or allowed. For example, in the absence of lookahead, the kernel can cover the current frame and the past L frames, such as two frames, and when L future frames are allowed, the kernel size can be 2L+1 centered on the current frame to match 2L+1 frames in the input data each time, such as 422, where L is two in 406. As Figure 4B As shown in 400B in FIG. 4, the lookahead can also be implemented with a series of conv2d layers 410, 412, or more layers. Each kernel then has a small kernel size along the time axis. For example, for 410, 412, and each other similar layer, L can be set to one. As a result, layer 410 can be matched with the original input data with 2L+1 lookahead, such as 422, where L is one leading to three kernels 428, and layer 412 can be matched with the output of layer 412. The server can use Figure 4B The illustrated series of conv2d layers gradually increases the receptive field within the input data.

[0063] In some embodiments, the number of cores in each conv2d layer may be determined based on the nature of the input audio stream, the volume of the desired high-level features, the extent of computational resource requirements, or other factors. For example, the number may be 8, 16, or 32. Additionally, each of the conv2d layers in block 308 may be followed by a nonlinear activation function, such as a parametric rectified linear unit (PReLU), which may then be followed by a separate batch normalization layer for fine-tuning the output of block 308.

[0064] In some embodiments, block 308 may be implemented using other signal processing techniques unrelated to artificial neural networks, such as the signal processing techniques described in C. Kim and R. M. Stern, “Power-Normalized Cepstral Coefficients (PNCC) for Robust Speech Recognition,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, Vol. 24, No. 7, pp. 1315-1329, July 2016, doi: 10.1109 / TASLP.2016.2545928.

[0065] 4.1.2.U-NET block

[0066] In some embodiments, Figure 3In block 340, server 102 encodes feature data (to find more and better features), then decodes to reconstruct enhanced audio data, and finally performs classification to determine how much speech is present. Therefore, block 340 includes an encoder side on the left and a decoder side on the right connected by block 350. The encoder includes one or more feature calculation blocks, such as 310, 312 and 314, each of which is followed by a frequency downsampler, such as 316, 318 and 320, to form a contraction path. A dense block (DB) is an embodiment of such a feature calculation block, as discussed further below. Each triple indicated in the figure, such as (8, T, 64), includes the size of the input or output data of the feature calculation block, wherein the first component represents the number of channels or feature maps, the second component represents the fixed number of frames along the time dimension, and the third component represents the size along the frequency dimension. As discussed further below, these feature calculation blocks capture higher and higher features in increasingly larger frequency contexts. Block 350 includes a feature calculation block for performing modeling covering all initially available perceptually motivated frequency bands. The decoder also includes one or more feature computation blocks, such as 320, 322, and 324, each of which is followed by a frequency upsampler, such as 326, 328, and 330, to form an expansion path. These feature computation blocks in the expansion path that rely on the feature maps generated during the contraction path are combined to project the distinguishing features at different levels (i.e., at each frequency band level at each frame) onto a high-resolution space to obtain a dense classification, i.e., a mask value. Due to the combination, the number of input channels (or feature maps) of each feature computation block in the expansion path can be twice the number of input channels (or feature maps) of each feature computation block in the contraction path. However, the choice of the number of cores in each computation block can determine the number of output channels, which becomes the number of input channels of the next feature computation block in the expansion path.

[0067] The server 102 generates a final mask value for each frequency band at a certain frame through a classification block (such as block 360), which includes a 1×1 2D kernel followed by a sigmoid nonlinear activation function.

[0068] In some embodiments, at each frequency downsampler, the server 102 merges every two adjacent band energies via a regular convolution or a depthwise convolution through a conv2d layer with a kernel size and a stride size of 2 along the frequency axis. Alternatively, the conv2d layer can be replaced by a maximum pooling layer. In either case, the width of the output feature map is halved after each frequency downsampler, thereby steadily increasing the receptive field within the input data. In order to achieve this continuous, exponential reduction in the width of the output feature map, the server 102 pads the output of block 308 to a width that is a power of 2, which then becomes the input data for block 340. For example, the padding can be accomplished by adding zeros to both sizes of the output feature map of block 308.

[0069] In some embodiments, in each frequency upsampler, the server 102 employs a transposed conv2d layer corresponding to a conv2d layer at the same level in the encoder to restore the original number of band energies. The depth of block 340 or the combined number of feature computation blocks and frequency downsamplers (and equivalently the combined number of feature computation blocks and frequency upsamplers) may depend on the desired maximum receptive field, the amount of computational resources, or other factors.

[0070] In some embodiments, the server 102 uses skip connections (e.g., 342, 344, and 346) to concatenate the output of a feature computation block in the encoder with the input of the same level of a feature computation block in the decoder as a way for the decoder to receive distinguishing features of different levels of input data that are ultimately used for dense classification, as described above. For example, the feature map produced by block 310 is concatenated with the feature map fed from the frequency upsampler 330 to block 324 via skip connection 346. Figure 1 As a result, the number of channels in the input data of each feature computation block in the decoder will be twice the number of channels in the input data of each dense block in the encoder.

[0071] In some embodiments, instead of direct concatenation, server 102 learns the scaler multipliers for each hop connection, such as α1, α2, and α3. Figure 3 As shown. Each α i Contains N (e.g., 8) learnable parameters that can be initialized to 1 at the beginning of training. Each learnable parameter is multiplied by the feature map produced by the corresponding feature calculation block in the encoder to produce a scaled feature map, which is then concatenated with the feature map of the corresponding feature calculation block to be fed to the decoder.

[0072] In some embodiments, the server 102 may use addition instead of concatenation. For example, the eight feature maps generated by block 310 may be added to the eight feature maps to be fed to dense block 324, where each of the eight additions is performed on a component basis. This addition instead of concatenation reduces the number of feature maps used as input data for each feature calculation block in the decoder and reduces computation overall at the expense of some performance degradation.

[0073] 4.1.2.1. Dense Blocks

[0074] Figure 5 An example neural network model is illustrated, which corresponds to Figure 3 The neural network model is based on a DenseNet structure, such as the DenseNet structure described in arXiv:1608.06993v5[cs.CV] on January 28, 2018, but with some changes, as described in this article. The DenseNet structure has been shown to alleviate the gradient vanishing problem, strengthen feature propagation, encourage feature reuse, and reduce the number of parameters.

[0075] In some embodiments, server 102 uses block 500 as a feature calculation block to further enhance feature propagation and dense classification. Block 500 outputs N (e.g., 8) feature map channels that are the same as the number of feature map input data. Each channel also has the same time-frequency shape as the feature map in the input data. Block 500 includes a series of convolutional layers, such as 520 and 530. The input data of each convolutional layer contains the concatenation of all output data of the previous convolutional layer, thereby forming a dense connection. For example, the input data of layer 530 includes data 512 (which can be initial input data or output data from a previous convolutional layer) and data 522 (which is input data from layer 520).

[0076] In some embodiments, each convolutional layer includes a bottleneck layer, such as layer 504, having one or more 1×1 2D kernels, for merging input data including K feature maps due to dense connections into a smaller number of feature maps. For example, each 1×1 2D kernel can be applied to each group of K / 2N feature maps separately to effectively add the K / 2N feature maps into one feature map and ultimately obtain 2N feature maps. Alternatively, a total of 2N 1×1 2D kernels can be applied to all feature maps to generate a 2D feature map. Each 1×1 2D kernel can be followed by a nonlinear activation function (such as PReLU) and / or a batch normalization layer.

[0077] In some embodiments, each convolutional layer includes a small conv2d layer with N kernels after the bottleneck layer, such as block 506 with a 3×3 convd2d layer, to produce N feature maps. These small conv2d layers in the consecutive convolutional layers of block 500 use exponentially growing dilation along the time axis to simulate increasingly larger context information. For example, the dilation factor used in block 506 is 1, meaning there is no dilation in each kernel, while the dilation factor used in block 508 is 2, meaning the kernel is dilated twice in time and the size of the receptive field is also increased twice in each dimension.

[0078] In some embodiments, between the convolutional layers of block 500, server 102 projects the band energies linearly to the learning space in the frequency mapping layer to obtain a more uniform output, as described in arXiv:1904.11148v1[cs.SD] April 25, 2019. Since the same kernel may have different effects on the same audio data depending on the frequency band in which the audio data is located, some uniformity of this effect across different frequency bands would be helpful. For example, frequency mapping layer 580 is located in the middle of the depth of block 500.

[0079] In some embodiments, at the end of block 500 , a layer 590 similar to a bottleneck layer having one or more 1×1 2D kernels may be used to produce an output tensor having N feature maps.

[0080] 4.1.1.1.1. Depthwise Separable Convolution with Gating

[0081] Figure 6 An example neural network model is illustrated, which corresponds to Figure 5 In some embodiments, block 600 includes a depthwise separable convolution with a nonlinear activation function, such as a gated linear unit (GLU). Figure 6 As shown, the first path in the GLU includes a conv2d layer of less depth, such as a 3×3 conv2d layer 602, followed by a batch normalization layer 604. The second path in the GLU similarly includes a 3×3 conv2d layer 606, followed by a batch normalization layer 608, and then a learnable gating function, such as a sigmoid nonlinear activation function. As in Figure 5In the illustrated dense block, the small conv2d layers in the successive convolutional layers of block 500 can use an exponentially growing dilation along the time axis to simulate increasingly larger context information. For example, blocks 602 and 606 corresponding to block 506 in the convolutional layer can be associated with a dilation factor of 1, and a similar block in the next convolutional layer that can correspond to an embodiment of block 508 can be associated with a dilation factor of 2. The gating function identifies important regions of the input data for the task of interest. The two paths are connected by a Hadamard product operator 618. As part of a depth-wise separable convolution, a 1×1 conv2d layer 612 learns the interconnections between the output feature maps generated by the combination of the two paths. Layer 612 can be followed by a batch normalization layer 614 and a non-linear activation function 616 (such as a PReLU).

[0082] 4.1.2.2 Residual Blocks and Recursive Layers

[0083] Figure 7 An example neural network model is illustrated, which corresponds to Figure 3 In some embodiments, the block 310 and each other similar block illustrated in FIG. Figure 5 The illustrated block 500 (also corresponding to an embodiment of block 310 ) may be replaced by a residual 700 block to reduce the number of connections. Block 700 includes a plurality of convolutional layers, such as layers 720 and 730 .

[0084] In some embodiments, each convolutional layer includes a layer similar to Figure 5 The bottleneck layer of block 504 is illustrated as layer 704. The bottleneck layer may also be followed by a non-linear activation (such as PReLU) and / or a batch normalization layer.

[0085] In some embodiments, the convolutional layer also includes a Figure 5 The illustrated block 506 has a small conv2d layer, such as a 3×3 conv2d layer 706. The small conv2d block can be performed with dilation, where the dilation factor increases exponentially on consecutive convolutional layers. Figure 6 As shown, the small conv2d layer can be replaced by a gated depthwise separable convolution.

[0086] In some embodiments, the convolutional layer includes another 1×1 conv2d layer, such as layer 708, which matches the output of block 706 back to the input of block 704 in terms of size, particularly the number of channels or feature maps. The output is then added to the input data via the Hadamard product operator 710 to reduce the gradient vanishing problem when backpropagation is used to train the network, as the gradient will have a direct path from the output to the input side without any multiplication in between. The conv1x1 layer can also be followed by a non-linear activation (such as PReLU) and / or a batch normalization layer.

[0087] In some embodiments, Figure 5 The illustrated block 500 (also corresponding to an embodiment of block 310) can be replaced by a recursive layer including at least one recursive neural network (RNN). Using RNN to simulate long time series can be an efficient method. "Efficient" means that the RNN can simulate very long time series by saving an internal hidden state vector as a summary of all the history it has seen and generating output for each new frame based on this vector. Compared with using expansion in the CNN layer, the buffer size used to store past information for the RNN is much smaller (only 1 vector, while there are 2d+1 vectors for CNN, where d is the expansion factor).

[0088] Model Training

[0089] In some embodiments, the training of the neural network model 208 can be performed as an end-to-end process. Alternatively, the feature extraction block 308 and the U-Net block 340 can be trained separately, where the output of applying the feature extraction block 308 to real data can be used as training data for the U-Net block.

[0090] Use different training data to train Figure 2 The illustrated neural network model 208. In some embodiments, diversity incorporates speaker diversity by including natural utterances of a wide range of speaking styles in terms of speed, emotion, and other attributes in the training data. Each training utterance can be speech from one speaker or a conversation between multiple speakers.

[0091] In some embodiments, diversity comes from the inclusion of centralized noise data (including reverberation data). Databases such as AudioSet can be used as seed noise databases. Server 102 can filter out each clip in the seed noise database with a category label indicating that speech may be present in the clip. For example, the "human voice" classification in a given ontology can be filtered out. The seed noise database can be further filtered by applying any speech separation technique known to those skilled in the art to remove additional clips where speech may be present. For example, any clips where speech prediction contains at least one frame (e.g., 100ms in length) with an RMS energy above a threshold (e.g., 1e-3) are removed.

[0092] In some embodiments, diversity is increased by including a wide range of intensity levels when mixing noise and speech. When synthesizing the noisy signal, the server 102 can scale the clean speech signal and the noise signal to a predetermined maximum sound level, randomly adjust each down by one dB in a dB range (e.g., 0 to 30 dB), and randomly add the adjusted clean speech signal and the adjusted noise signal, subject to a predetermined minimum signal-to-noise ratio. It has been found that this wide range of loudness levels helps reduce over-suppression of speech (or under-suppression of noise).

[0093] In some embodiments, the diversity lies in the presence of data in different frequency bands. Server 102 may create a signal having at least a certain percentage in a specific frequency band of a specific bandwidth, such as at least 20% in the frequency band of 300 Hz to 500 Hz.

[0094] In some embodiments, the server 102 uses any optimization process known to those skilled in the art (such as a stochastic gradient descent optimization algorithm that uses an error back-propagation algorithm to update weights) to train the neural network model 208. The neural network model 208 can minimize the mean square error (MSE) loss between the predicted mask and the true data mask for each frequency band at each frame. The true data mask can be calculated as the ratio of the speech energy to the sum of the speech and noise energies.

[0095] In some embodiments, because over-suppression of speech is more detrimental to speech quality than under-suppression of speech, server 102 uses a weighted MSE that assigns more penalty to over-suppression of speech. Since the mask value generated by neural network model 208 indicates the amount of speech present, when the predicted mask value is less than the true data mask value, less speech is predicted than the true data, and therefore more speech is suppressed than necessary, resulting in over-suppression of speech by the neural network model. For example, the weighted MSE can be calculated as follows:

[0096]

[0097] in, and m(t, f) represent the predicted mask value and the true data mask value of the (t, f) time-frequency band, respectively, and ρ represents an empirically determined constant (usually set to greater than 0.5) to give greater weight to speech over-suppression.

[0098] In some embodiments, the neural network model 208 is trained to predict the distribution of speech over different frequency bins within each frequency band (rather than a single mask value). Specifically, the server 102 can train the model to predict the mean and variance values ​​of the Gaussian distribution for each frequency band at each frame, where the mean represents the best prediction of the mask value by the neural network model 208. The loss function for the Gaussian distribution can be defined as:

[0099]

[0100] in, represents the prediction of the standard deviation of (t, f).

[0101] In some embodiments, the variance prediction can be interpreted as a confidence level in the mean prediction to reduce the occurrence of over-suppression of speech. When the mean prediction value is relatively low, it indicates that there is a small amount of speech, while when the variance prediction value is relatively high, this may indicate that speech may be over-suppressed, and the band mask can then be scaled up. An example scaling function that produces an adjusted gain based on the standard deviation is:

[0102]

[0103] The scaling function increases the band mask (gain) in proportion to the standard deviation. When the standard deviation is large, the mask is scaled so that it is greater than the mean but still less than or equal to 1, and when the standard deviation is 0, the mask will be equal to the mean.

[0104] In some embodiments, assuming a Gaussian distribution for each mask, the probability of each observed (target) mask value is:

[0105]

[0106] Minimizing the negative logarithm of this probability (equivalent to maximizing the probability itself) yields the Gaussian loss function described above.

[0107] Model Execution

[0108] In some embodiments, when implementing look-ahead in the neural network model 208, in particular the feature extraction block 308, the server 102 may accept a single frame or a group of frames as input data and generate at least a mask value for each frame as output data. For each convolutional layer with a kernel size greater than one along the time dimension, the server 102 maintains an internal buffer to store the history required to generate the output data. The buffer may be maintained as a queue whose size is equal to the receptive field of the convolutional layer along the time dimension.

[0109] 5. Example Process

[0110] Figure 8 An example process performed by an audio management server computer according to some embodiments described herein is illustrated. Figure 8 The examples are shown in a simplified schematic format for purposes of illustrating clarity, and other embodiments may include more, fewer, or different elements connected in various ways. Figure 8Each is intended to disclose an algorithm, plan or outline that can be used to implement one or more computer programs or other software elements that, when executed, enable the functional improvements and technical advances described herein to be performed. In addition, the flowcharts herein are described with the same level of detail that ordinary technicians in the field typically use to communicate with each other about the algorithms, plans or specifications that form the basis of the software programs they plan to write or implement using their accumulated skills or knowledge.

[0111] In some embodiments, in step 802, the server 102 is programmed to receive input audio data covering multiple frequency bands along the frequency dimension at multiple frames along the time dimension. In some embodiments, the multiple frequency bands are perceptually motivated frequency bands, covering more frequency bins at higher frequencies.

[0112] In some embodiments, in step 804, the server 102 is programmed to train a neural network model. The neural network model includes: a feature extraction block that implements a look-ahead of a particular number of frames when extracting features from the input audio data; an encoder that includes a first series of blocks that produce feature maps corresponding to increasingly larger receptive fields along a frequency dimension in the input audio data; a decoder that includes a second series of blocks that receive as input feature maps the output feature maps generated by the encoder; and a classification block that generates a speech value that indicates an amount of speech present for each of a plurality of frequency bands at each of a plurality of frames.

[0113] In some embodiments, the feature extraction block has a convolution kernel having a specific size along the time dimension, and the encoder and the decoder do not have a convolution kernel having a size equal to or greater than the specific size along the time dimension. In other embodiments, each of the feature extraction block, the first series of blocks, and the second series of blocks generates a common number of feature maps.

[0114] In some embodiments, the feature extraction block includes a batch normalization layer followed by a convolutional layer with a two-dimensional convolutional kernel.

[0115] In some embodiments, each block in the first series of blocks in the encoder includes a feature computation block and a frequency downsampler. The feature computation block includes a series of convolutional layers.

[0116] In some embodiments, output data of a convolutional layer in the series of convolutional layers is fed into all subsequent convolutional layers in the series of convolutional layers. The series of convolutional layers implements increasingly larger dilation along the time dimension. In other embodiments, each of the series of convolutional layers comprises a depthwise separable convolutional block with a gating mechanism.

[0117] In some embodiments, each of the series of convolutional layers includes a residual block having a series of convolutional blocks, including a first convolutional block having a first one-by-one two-dimensional convolutional kernel and a last convolutional block having a last one-by-one two-dimensional convolutional kernel.

[0118] In some embodiments, output data of a feature computation block in a block in the first series of blocks is scaled by a learnable weight to form scaled output data, and the scaled output data is transmitted to a block in the second series of blocks in the decoder via a skip connection.

[0119] In some embodiments, the frequency downsampler of a block in the first series of blocks comprises a convolution kernel having a stride size greater than one along the frequency dimension.

[0120] In some embodiments, each block in the second series of blocks includes a feature calculation block and a frequency upsampler. The feature calculation block in a block in the second series of blocks receives first output data from a feature calculation block in a block in the first series of blocks and second output data from a frequency upsampler of a previous block in the second series of blocks. The first output data and the second output data are then concatenated or added to form specific input data for the feature calculation block in the block in the second series of blocks.

[0121] In some embodiments, the classification block includes a one-by-one two-dimensional convolution kernel and a non-linear activation function.

[0122] In some embodiments, the neural network model further includes a feature calculation block that serves as output data of the encoder and input data of the decoder.

[0123] In some embodiments, the server 102 is programmed to perform training using a loss function between the predicted speech value and the real data speech value for each of the plurality of frequency bands at each frame, wherein the weight in the loss function is larger when the predicted speech value corresponds to over-suppression of speech, and the weight in the loss function is smaller when the predicted speech value corresponds to under-suppression of speech. In some embodiments, the classification block further generates a distribution of speech volume on a frequency band in the plurality of frequency bands at a certain frame, wherein the speech value is the mean of the distribution.

[0124] In some embodiments, the input audio data includes data corresponding to speech of different speeds or emotions, data containing different levels of noise, or data corresponding to different frequency bins.

[0125] In some embodiments, in step 806, server 102 is programmed to receive new audio data comprising one or more frames.

[0126] In some embodiments, in step 808, server 102 is programmed to execute the neural network model on the new audio data to generate new speech values ​​for each of the plurality of frequency bands at each of the one or more frames.

[0127] In some embodiments, in step 810, server 102 is programmed to generate new output data that suppresses noise in the new audio data based on the new speech value.

[0128] In some embodiments, in step 812, server 102 is programmed to transmit new output data.

[0129] In some embodiments, the server 102 is programmed to receive an input waveform. The server 102 is programmed to then transform the input waveform into raw audio data, which covers multiple frequency bins along the frequency dimension at one or more frames along the time dimension. The server 102 is programmed to then convert the raw audio data into new audio data by grouping multiple frequency bins into multiple frequency bands. The server 102 is programmed to perform inverse striping on the new voice value to generate an updated voice value for each frequency bin in the multiple frequency bins at each frame in one or more frames. In addition, the server 102 is programmed to then apply the updated voice value to the raw audio data to generate new output data. Finally, the server 102 is programmed to transform the new output data into an enhanced waveform.

[0130] 6. Hardware Implementation

[0131] According to one embodiment, the technology described herein is implemented by at least one computing device. These technologies can be implemented in whole or in part using a combination of at least one server computer and / or other computing devices coupled using a network (such as a packet data network). The computing device can be hard-wired to perform these technologies, or can include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field programmable gate array (FPGA) that is permanently programmed to perform these technologies, or can include at least one general-purpose hardware processor that is programmed to perform these technologies according to program instructions in firmware, memory, other storage devices, or combinations. Such computing devices can also combine customized hard-wired logic, ASIC or FPGA with customized programming to implement these described technologies. The computing device can be a server computer, a workstation, a personal computer, a portable computer system, a handheld device, a mobile computing device, a wearable device, a body-mounted or implantable device, a smart phone, a smart appliance, a network interconnected device, an autonomous or semi-autonomous device such as a robot or an unmanned ground or air vehicle, any other electronic device that incorporates hard wiring and / or program logic to implement the described technology, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0132] Fig. 9 is a block diagram illustrating an example computer system that may be used to implement embodiments. Fig. 9 In the example of , a computer system 900 and instructions for implementing the disclosed techniques in hardware, software, or a combination of hardware and software are schematically represented, for example, as boxes and circles, with the same level of detail that is commonly used by ordinary technicians in the field to which the present disclosure belongs to communicate about computer architectures and computer system implementations.

[0133] The computer system 900 includes an input / output (I / O) subsystem 902, which may include a bus and / or other communication mechanism(s) for transferring information and / or instructions between components of the computer system 900 via electronic signal paths. The I / O subsystem 902 may include an I / O controller, a memory controller, and at least one I / O port. Electronic signal paths are schematically represented in the figures as, for example, lines, unidirectional arrows, or bidirectional arrows.

[0134] At least one hardware processor 904 is coupled to the I / O subsystem 902 for processing information and instructions. The hardware processor 904 may include, for example, a general purpose microprocessor or microcontroller and / or a dedicated microprocessor such as an embedded system or a graphics processing unit (GPU) or a digital signal processor or an ARM processor. The processor 904 may include an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

[0135] The computer system 900 includes one or more units of memory 906, such as main memory, which is coupled to the I / O subsystem 902 to electronically digitally store data and instructions to be executed by the processor 904. The memory 906 may include volatile memory (such as various forms of random access memory (RAM)) or other dynamic storage devices. The memory 906 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 904. When stored in a non-transitory computer-readable storage medium accessible to the processor 904, such instructions can turn the computer system 900 into a special-purpose machine customized to perform the operations specified in the instructions.

[0136] The computer system 900 further includes non-volatile memory such as read-only memory (ROM) 908 or other static storage devices coupled to the I / O subsystem 902 to store information and instructions for the processor 904. The ROM 908 may include various forms of programmable ROM (PROM) such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). The persistent storage unit 910 may include various forms of non-volatile RAM (NVRAM) such as flash memory or solid-state storage devices, magnetic disks or optical disks such as CD-ROMs or DVD-ROMs, and may be coupled to the I / O subsystem 902 for storing information and instructions. The storage device 910 is an example of a non-transitory computer-readable medium that may be used to store instructions and data that, when executed by the processor 904, enable computer-implemented methods for performing the techniques herein to be performed.

[0137] The instructions in the memory 906, ROM 908 or storage device 910 may include one or more groups of instructions organized as modules, methods, objects, functions, routines or calls. The instructions may be organized as one or more computer programs, operating system services or applications including mobile applications. The instructions may include operating systems and / or system software; one or more libraries supporting multimedia, programming or other functions; data protocol instructions or stacks for implementing TCP / IP, HTTP or other communication protocols; file processing instructions for interpreting and presenting files encoded using HTML, XML, JPEG, MPEG or PNG; user interface instructions for presenting or interpreting commands for graphical user interfaces (GUIs), command line interfaces or text user interfaces; application software such as office suites, Internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or other applications. These instructions may implement a web server, a web application server or a web client. The instructions may be organized into a presentation layer, an application layer and a data storage layer such as a relational database system using structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system or other data store.

[0138] The computer system 900 can be coupled to at least one output device 912 via the I / O subsystem 902. In one embodiment, the output device 912 is a digital computer display. Examples of displays that can be used in various embodiments include a touch screen display or a light emitting diode (LED) display or a liquid crystal display (LCD) or an electronic paper display. The computer system 900 may include other (multiple) types of output devices 912 as an alternative or supplement to the display device. Examples of other output devices 912 include a printer, a ticket printer, a plotter, a projector, a sound card or a video card, a speaker, a buzzer or a piezoelectric device or other audible device, a lamp or an LED or LCD indicator, a tactile device, an actuator or a servo.

[0139] At least one input device 914 is coupled to the I / O subsystem 902 for communicating signals, data, command selections, or gestures to the processor 904. Examples of input device 914 include touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, sliders, and / or various types of sensors such as force sensors, motion sensors, thermal sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors and / or various types of transceivers such as wireless (e.g., cellular or Wi-Fi) transceivers, radio frequency (RF) transceivers, or infrared (IR) transceivers and global positioning system (GPS) transceivers.

[0140] Another type of input device is a control device 916, which can perform cursor control or other automatic control functions, such as navigating in a graphical interface on a display screen, as a substitute or supplement for the input function. The control device 916 can be a touch pad, a mouse, a trackball, or a cursor direction key, for transmitting direction information and command selections to the processor 904 and for controlling the movement of the cursor on the display 912. The input device can have at least two degrees of freedom on two axes (a first axis (e.g., an x-axis) and a second axis (e.g., a y-axis)), allowing the device to specify an orientation in a plane. Another type of input device is a wired control device, a wireless control device, or an optical control device, such as a joystick, a stick, a console, a steering wheel, a pedal, a shift mechanism, or other types of control devices. The input device 914 can include a combination of multiple different input devices, such as a camera and a depth sensor.

[0141] In another embodiment, the computer system 900 may include an Internet of Things (IoT) device in which one or more of the output device 912, the input device 914, and the control device 916 are omitted. Alternatively, in such an embodiment, the input device 914 may include one or more cameras, motion detectors, thermometers, microphones, earthquake detectors, other sensors or detectors, measurement devices, or encoders, and the output device 912 may include a dedicated display such as a single-line LED or LCD display, one or more indicators, display panels, meters, valves, solenoids, actuators, or servos.

[0142] When the computer system 900 is a mobile computing device, the input device 914 may include a global positioning system (GPS) receiver coupled to a GPS module capable of triangulating a plurality of GPS satellites, determining and generating a geographic location or location data, such as a latitude-longitude value for the geophysical location of the computer system 900. The output device 912 may include hardware, software, firmware, and interfaces for generating, alone or in combination with other application-specific data, position report packets, notifications, pulse or heartbeat signals, or other repetitive data transmissions specifying the location of the computer system 900 directed to the host computer 924 or server 930.

[0143] The computer system 900 may implement the techniques described herein using custom hardwired logic, at least one ASIC or FPGA, firmware, and / or program instructions or logic that, when loaded and used or executed, combine with the computer system to cause the computer system to operate as a special purpose machine. According to one embodiment, the computer system 900 performs the techniques herein in response to the processor 904 executing at least one sequence of at least one instruction contained in the main memory 906. Such instructions may be read into the main memory 906 from another storage medium, such as the storage device 910. Execution of the sequence of instructions contained in the main memory 906 causes the processor 904 to perform the process steps described herein. In alternative embodiments, hardwired circuits may be used in place of or in combination with software instructions.

[0144] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 910. Volatile media include dynamic memory, such as memory 906. Common forms of storage media include, for example, hard disks, solid-state drives, flash drives, magnetic data storage media, any optical or physical data storage media, memory chips, etc.

[0145] Storage media are distinct from transmission media but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the wires of bus 902 that contain the I / O subsystem. Transmission media can also take the form of acoustic or light waves, such as those generated during wireless and infrared data communications.

[0146] Various forms of media may be involved in carrying at least one sequence of at least one instruction to the processor 904 for execution. For example, the instruction may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instruction into its dynamic memory and use a modem to send these instructions through a communication link such as an optical fiber or coaxial cable or a telephone line. A modem or router local to the computer system 900 may receive data on the communication link and convert the data for reading by the computer system 900. For example, a receiver such as a radio frequency antenna or an infrared detector may receive data carried in a wireless or optical signal, and appropriate circuitry may provide the data to the I / O subsystem 902, such as placing the data on a bus. The I / O subsystem 902 carries the data to the memory 906, from which the processor 904 obtains and executes the instruction. The instruction received by the memory 906 may optionally be stored on the storage device 910 before or after being executed by the processor 904.

[0147] The computer system 900 also includes a communication interface 918 coupled to the bus 902. The communication interface 918 provides a two-way data communication coupling to (multiple) network links 920, which are directly or indirectly connected to at least one communication network such as a network 922 or a public or private cloud on the Internet. For example, the communication interface 918 can be an Ethernet network interface, an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection to a corresponding type of communication line (e.g., an Ethernet cable or any kind of metal cable or fiber optic line or telephone line). The network 922 broadly represents a local area network (LAN), a wide area network (WAN), a campus network, the Internet, or any combination thereof. The communication interface 918 can include a LAN card that provides a data communication connection to a compatible LAN, or a cellular radiotelephone interface that sends or receives cellular data by wire according to a cellular radiotelephone wireless network standard, or a satellite radio interface that sends or receives digital data by wire according to a satellite wireless network standard. In any such implementation, communication interface 918 sends and receives electrical, electromagnetic or optical signals through the signal path that carry digital data streams representing various types of information.

[0148] Network link 920 typically provides electrical, electromagnetic, or optical data communication to other data devices directly or through at least one network using, for example, satellite, cellular, Wi-Fi, or Bluetooth technology. For example, network link 920 may provide a connection to a host computer 924 through network 922.

[0149] In addition, network link 920 can provide a connection through network 922 or a connection to other computing devices via Internet devices and / or computers operated by an Internet Service Provider (ISP) 926. ISP 926 provides data communication services through a worldwide packet data communication network (represented as Internet 928). Server computer 930 can be coupled to Internet 928. Server 930 broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or a computer that executes a containerized program system such as DOCKER or Kubernetes. Server 930 can represent an electronic digital service that is implemented using more than one computer or instance and accessed and used by transmitting a web service request, a uniform resource locator (URL) string with parameters in an HTTP payload, an API call, an application service call, or other service calls. Computer system 900 and server 930 can form an element of a distributed computing system that includes other computers, processing clusters, server groups, or other computer organizations that collaborate to perform tasks or execute applications or services. Server 930 may include one or more groups of instructions organized as modules, methods, objects, functions, routines, or calls. Instructions may be organized as one or more computer programs, operating system services, or applications including mobile applications. Instructions may include operating systems and / or system software; one or more libraries supporting multimedia, programming, or other functions; data protocol instructions or stacks for implementing TCP / IP, HTTP, or other communication protocols; file format processing instructions for interpreting or presenting files encoded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for presenting or interpreting commands for graphical user interfaces (GUIs), command line interfaces, or text user interfaces; application software such as office suites, Internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or other applications. Server 930 may include a web application server hosting a presentation layer, an application layer, and a data storage layer such as a relational database system using structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0150] Computer system 900 may send messages and receive data and instructions, including program code, through the network(s), network link 920, and communication interface 918. In the Internet example, server 930 may transmit the requested code for an application program through Internet 928, ISP 926, local network 922, and communication interface 918. The received code may be executed by processor 904 as it is received, and / or stored in storage device 910, or other non-volatile storage, for later execution.

[0151] The execution of the instructions described in this section can implement a process in the form of a computer program instance that is executed and consists of program code and its current activity. According to the operating system (OS), a process can be composed of multiple execution threads that execute instructions simultaneously. In this context, a computer program is a passive collection of instructions, and a process can be the actual execution of these instructions. Several processes can be associated with the same program; for example, opening several instances of the same program usually means executing more than one process. Multitasking can be implemented to allow multiple processes to share the processor 904. Although each processor 904 or the core of the processor executes a single task at a time, the computer system 900 can be programmed to implement multitasking to allow each processor to switch between the tasks being executed without having to wait for each task to complete. In an embodiment, switching can be performed when the task performs an input / output operation, when the task indicates that it can be switched, or when a hardware interrupt is performed. By quickly performing context switching to allow multiple processes to be executed simultaneously, time-sharing can be implemented to allow the rapid response of interactive user applications. In an embodiment, for security and reliability, the operating system can prevent direct communication between independent processes, providing a strictly mediated and controlled inter-process communication function.

[0152] 7. Extensions and Alternatives

[0153] In the foregoing specification, embodiments of the present disclosure have been described with reference to numerous specific details, which may vary depending on the implementation. Therefore, the specification and drawings should be viewed in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the present disclosure and the applicant's inventive intent within the scope of the present disclosure are the claims issued from this application in the specific form in which the claims issue, including any subsequent amendments.

Claims

1. A computer-implemented method for suppressing noise and enhancing speech, the method comprising: receiving, by a processor, input audio data covering a plurality of frequency bands along a frequency dimension at a plurality of frames along a time dimension; The processor trains a neural network model using the input audio data, the neural network model comprising: a feature extraction block that implements a look-ahead of a specified number of frames when extracting features from the input audio data; an encoder comprising a first series of blocks producing a first feature map corresponding to increasingly larger receptive fields in the input audio data along the frequency dimension; a decoder comprising a second series of blocks that receive as input the output feature map generated by the encoder and produce a second feature map; and a classification block that receives the second feature map and generates a speech value that indicates an amount of speech present for each of the plurality of frequency bands at each of the plurality of frames; receiving new audio data comprising one or more frames; executing the neural network model on the new audio data to generate new speech values ​​for each of the plurality of frequency bands at each of the one or more frames; generating new output data suppressing noise in the new audio data based on the new speech value; The new output data is transmitted.

2. The computer-implemented method of claim 1 , further comprising: receiving an input waveform; transforming the input waveform into raw audio data covering a plurality of frequency bins along the frequency dimension at the one or more frames along the time dimension; converting the original audio data into the new audio data by grouping the plurality of frequency bins into the plurality of frequency bands; performing inverse striping on the new speech value to generate an updated speech value for each of the plurality of frequency bins at each of the one or more frames; applying the updated speech value to the original audio data to generate the new output data; The new output data is transformed into an enhanced waveform.

3. The computer-implemented method of one of claims 1 and 2, wherein the plurality of frequency bands are perceptually motivated frequency bands covering more frequency bins at higher frequencies.

4. A computer-implemented method as claimed in any one of claims 1 to 3, The feature extraction block has a convolution kernel, and the convolution kernel has a specific size along the time dimension, The specific size is larger than the size of any convolution kernel in the encoder or the decoder along the time dimension.

5. The computer-implemented method of any one of claims 1 to 4, the feature extraction block comprising a batch normalization layer followed by a convolutional layer with a two-dimensional convolutional kernel.

6. The computer-implemented method of any one of claims 1 to 5, each of the feature extraction block, the first series of blocks, and the second series of blocks generating a common number of feature maps.

7. A computer-implemented method as claimed in any one of claims 1 to 6, Each block in the first series of blocks includes a feature calculation block and a frequency downsampler, The feature calculation block includes a series of convolutional layers.

8. The computer-implemented method of claim 7, The output data of a convolutional layer in the series of convolutional layers is fed into all subsequent convolutional layers in the series of convolutional layers, The series of convolutional layers implements increasingly larger dilation along the time dimension.

9. A computer system comprising: Memory; One or more processors coupled to the memory and configured to perform the following operations: receiving input audio data covering a plurality of frequency bands along a frequency dimension at a plurality of frames along a time dimension; The neural network model is trained using the input audio data, the neural network model comprising: a feature extraction block that implements a look-ahead of a specified number of frames when extracting features from the input audio data; an encoder comprising a first series of blocks producing a first feature map corresponding to increasingly larger receptive fields in the input audio data along the frequency dimension; a decoder comprising a second series of blocks that receive as input the output feature map generated by the encoder and produce a second feature map; and a classification block that receives the second feature map and generates a speech value that indicates an amount of speech present for each of the plurality of frequency bands at each of the plurality of frames; The neural network model is stored.

10. A computer-implemented method for suppressing noise and enhancing speech, the method comprising: receiving, by a processor, new audio data comprising one or more frames; executing, by the processor, a neural network model on the new audio data to generate new speech values ​​for each of a plurality of frequency bands at each of the one or more frames; The neural network model includes computer executable instructions for: a feature extraction block that implements a look-ahead of a certain number of frames when extracting features from the input audio data; an encoder comprising a first series of blocks producing a first feature map corresponding to increasingly larger receptive fields in the input audio data along the frequency dimension; A computing block, the computing block connecting the encoder and the decoder; the decoder comprising a second series of blocks that receive as input feature maps the output feature maps generated by the encoder and produce second feature maps; as well as a classification block that receives the second feature map and generates a speech value that indicates an amount of speech present for each of the plurality of frequency bands at each of a plurality of frames; as well as training the neural network model with the input audio data, the input audio data covering the plurality of frequency bands along the frequency dimension at the plurality of frames along the time dimension; generating new output data suppressing noise in the new audio data based on the new speech value; The new output data is transmitted.

Images