Machine learning model architecture for a language improvement system

Abstract

Methods and systems for a machine learning model architecture for a speech improvement system. The disclosed machine learning model architecture includes, among other things, an encoder, a decoder, and a bottleneck located between the encoder and the decoder. The encoder contains multiple encoder layers, while the decoder contains multiple decoder layers. Each encoder layer is connected to a corresponding decoder layer via a jump link. Each encoder layer contains a Res2Net and a compression and excitation (SE) block. The bottleneck contains a first layer of controlled recurrent units (GRU layer) and a second GRU layer.

Description

Aspects and implementations of the present disclosure relate to a machine learning model architecture for a speech enhancement system.Speech enhancement systems have become important in enhancing the perceptual quality and intelligibility of speech waveforms contaminated by additive background noise and Hall.In this respect, it is an object to improve known approaches.This object is achieved according to the features of the independent claims. Preferred embodiments can be found in particular in the dependent claims.These examples proposed herein can be based in particular on at least one of the following solutions. In particular, combinations of the following features may be employed to achieve a desired result. The features of the method, the computer readable medium, or the system may be combined with any feature(s) of any of the other categories.For the solution, for example, a process is specified, comprising the steps:receiving a first speech waveform;processing an input based on the first speech waveform using a trained machine learning model to generate a second speech waveform, the trained machine learning model comprising:an encoder, wherein the encoder comprises a plurality of encoder layers, each encoder layer comprising:receiving, by a residue network block of a respective encoder layer, as input one of: a first speech waveform or multiscale feature maps output from a previous encoder layer of the plurality of encoder layers;generating, based on the input, multiscale feature maps of the respective encoder layer.In one development, the residual network block is a Res2Net.In one embodiment, each encoder layer further comprises:receiving an output of the residual network block of the respective encoder layer through a compression and excitation, SE, block of the respective encoder layer;adaptive recalibration of the multi-scale feature maps based on the received output of the residual network block of the respective encoder layer.In one embodiment, each encoder layer further comprises:receiving an output of the SE block of the respective encoder layer by an extended convolution of the respective encoder layer;enlarging, by the extended convolution of the respective encoder layer, the recording fields of the multiscale feature maps without loss of the spatial resolution;receiving, by a rectified linear unit, ReLU, activation function, activation function of the respective encoder layer, an output of the extended convolution of the respective encoder layer;introducing non-linearity by the ReLU activation function of the respective encoder layer;receiving an output of the ReLU activation function of the respective encoder layer through a stack normalization, BN, of the respective encoder layer;normalizing an output of the ReLU activation function of the respective encoder layer by the stack normalization of the respective encoder layer.In one embodiment, the trained machine learning model further comprises:a bottleneck disposed between the encoder and a decoder of the machine learning model, the bottleneck comprising:receiving an output of the encoder through a first layer of controlled recurrent units, GRU layer;performing, by the first GRU layer, a first non-linear transformation;receiving an output of the first GRU layer through a second GRU layer;performing a second non-linear transformation by the second GRU layer.In one development, the first GRU layer and the second GRU layer are a unidirectional GRU layer.In one embodiment, the trained machine learning model further comprises:a decoder, wherein the decoder comprises a decoder of the encoder layers;a plurality of hop connections, each hop connection connecting a decoder layer of the decoder layers to an encoder layer of the encoder.It is a refinement that the first speech waveform is substantially equivalent to a second speech waveform that includes at least one of: a convolution space impulse response, convolution RIR, and additive background noise.Further proposed is a non-transitory computer readable medium comprising instructions that, in response to execution by a processing device, cause the processing device to perform operations comprising:receiving a plurality of speech waveform pairs, each speech waveform pair of the plurality of speech waveform pairs including a first speech waveform and a second speech waveform;training a machine learning model to generate a third speech waveform based on the first speech waveform of a respective speech waveform, wherein training the machine learning model comprises:inputting the first speech waveform of the respective speech waveform into the machine learning model;predicting the third speech waveform based on the input first speech waveform;inputting the second speech waveform of the respective speech waveform and the third speech waveform into a loss function;optimizing the model of the machine learning based on a result of the loss function, wherein the result of the loss function is calculated based on(i) a difference between the second speech waveform of the respective speech waveform and the third speech waveform; and(ii) a difference between a magnitude spectrogram of the second speech waveform of the respective speech waveform and a magnitude spectrogram of the third speech waveform.In one development, the first speech waveform of the respective speech waveform is the second speech waveform of the respective speech waveform, comprising at least one of the following: a convolution space impulse response, convolution RIR, and additive background noise.In one development, the machine learning model comprises the following:an encoder, wherein the encoder comprises a plurality of encoder layers, each encoder layer comprising:a residue net block, the residue net block receiving one of: the first speech waveform of the respective speech waveform or multiscale feature maps output from a previous encoder layer of the plurality of encoder layers to generate multiscale feature maps of a respective encoder layer.In one development, the residual network block is a Res2Net.In one embodiment, each encoder layer further comprises:a compression and excitation, SE, block, wherein the SE block receives an output of the residual network block of the respective encoder layer and adaptively recalibrations the multi-scale feature maps.In one embodiment, each encoder layer further comprises:an extended convolution, wherein the extended convolution receives an output signal of the SE block of the respective encoder layer and increases the recording fields of the multiscale feature maps without losing the spatial resolution;an ReLU Rectified Linear Unit, Activation, function, wherein the ReLU activation function receives the extended convolution output of the respective encoder layer and introduces a nonlinearity;a batch normalization, BN, wherein the batch normalization receives the output of the ReLU activation function of the respective encoder layer and normalizes the output of the ReLU activation function of the respective encoder layer.In one embodiment, the machine learning model further comprises:a bottleneck disposed between an encoder and a decoder of the machine learning model, the bottleneck comprising:a first layer of controlled recurrent units, GRU, layer, said first GRU layer receiving an output of said encoder and performing a first non-linear transformation;a second GRU layer, wherein the second GRU layer receives an output of the first GRU layer and performs a second non-linear transformation.Also provided is a system comprising:a storage device; anda processing device coupled to the memory device, the processing device to perform operations comprising:receiving a first speech waveform by a speech enhancement system;generating a latent output of the first speech waveform by an encoder of the speech enhancement system, wherein each layer of a plurality of encoder layers of the encoder comprises a residue network block and a compression and excitation, SE, block;executing, by a bottleneck of the speech enhancement system, a non-linear transform on the latent output of the first speech waveform, the bottleneck comprising a first layer of controlled recurrent units, GRU layer, and a second GRU layer;predicting, by a decoder of the speech enhancement system, a second speech waveform from the non-linear transformation of the latent output of the first speech waveform.It is a refinement that the first speech waveform is substantially equivalent to a second speech waveform that includes at least one of: a convolution space impulse response, convolution RIR, and additive background noise.In one development, the generation of the latent output of the first speech signal form comprises the following:for each encoder layer of the plurality of encoder layers, generating, by the residue network block of a respective encoder layer, multi-scale feature maps associated with an input of the respective encoder layer,an adaptive recalibration of the multi-scale feature maps by the SE block of the respective encoder layer, wherein an output of the respective encoder layer is a latent output of the input of the respective encoder layer.In one embodiment, the execution of the nonlinear transformation on the latent output of the first speech waveform by the bottleneck of the speech enhancement system includes the following:receiving the latent output of the first speech waveform;performing, by the first GRU layer, a first non-linear transform on the latent output of the first speech waveform;receiving an output of the first GRU layer through the second GRU layer;performing, by the second GRU layer, a second non-linear transformation.In one development, predicting the second speech waveform from the nonlinear transformation of the latent output of the first speech waveform comprises the following:receiving, for each decoder layer of a plurality of decoder layers of the decoder, a latent output of an encoder layer of the plurality of encoder layers connected to a respective decoder layer via a hopping connection.The aspects and implementations of the present disclosure will be more fully understood from the detailed description given below and the accompanying drawings of various aspects and implementations of the disclosure, which, however, should not be construed to limit the disclosure to the specific aspects or implementations, but are for explanation and understanding only. FIG. 1 illustrates an example speech enhancement system in accordance with implementations of the present disclosure. FIG. 2 is an exemplary illustration of a machine learning model architecture used for speech enhancement, in accordance with implementations of the present disclosure. FIGS. 3A and 3B are an exemplary illustration of an encoder of the machine learning model of FIG. 2, in accordance with implementations of the present disclosure. FIG. 4 is an exemplary illustration of a Res2Net layer of an encoder layer of the encoder of FIG. 3, in accordance with implementations of the present disclosure. FIG. 5 is an exemplary illustration of a compression and excitation (SE) layer of an encoder layer of the encoder of FIG. 3, in accordance with implementations of the present disclosure. FIG. 6 is an example illustration of a bottleneck of the machine learning model of FIG. 2, according to implementations of the present disclosure. FIG. 7 is an exemplary illustration of training the machine learning model of FIG. 2, in accordance with implementations of the present disclosure. FIG. 8 illustrates a flow diagram of an example method for processing a waveform using the speech enhancement system, in accordance with implementations of the present disclosure. FIG. 9 is a block diagram illustrating an example computer system in accordance with implementations of the present disclosure.Aspects of the present disclosure relate to a machine learning model architecture for a speech enhancement system. Speech enhancement systems are typically designed to enhance the quality and intelligibility of speech signals, particularly in noise or other form of interference environments. Speech enhancement systems are critical in various applications, e.g. speech recognition, hearing aids and modern telecommunication systems.Conventional speech enhancement systems use one or more methods to achieve better noise suppression and speech quality. Such speech enhancement systems, such as a speech enhancement system using spectral subtraction in conjunction with Wiener filtering, produce an estimate of the distribution or composition of the frequencies present in the noise of the speech signal (e.g., noise spectra) and / or the adjustment to an amplitude of the speech signal (e.g., gain mask). Thus, the speech enhancement system multiplies the estimated gain mask by the noisy spectrum of the speech signal to generate the enhanced spectrum. Conventional speech enhancement systems operate assuming that the noise is stationary or slowly varying, such as constant background hum, fan noise and slowly varying traffic noise. In real world environments, noise is often not stationary, changing rapidly over time. Non-stationary sounds include, for example, background conversations, doors to be turned on, hurking cars, dog bells, bird's-twitches, taps on the keyboard, etc. Consequently, conventional speech enhancement systems tend to fail to suppress the noise and enhance speech quality. This is further complicated by the low signal-to-noise ratio (SNR) conditions, in which the noise level is high compared to the speech level, so that it is difficult to distinguish between noise and speech, resulting in a less accurate estimation.In recent years, machine learning and deep learning algorithms are increasingly used because they are more convenient in handling non-stationary noise and keeping speech quality. Most important is the implementation of deep neural networks (DNNs) in speech enhancement systems (e.g., DNN speech enhancement systems). The DNN speech enhancement systems have significantly advanced the art and provide more robust performance than conventional speech enhancement systems under various noise conditions including non-stationary noise conditions and low SNR. The DNN speech enhancement systems are typically time-frequency based or waveform domain based.The time-frequency-based DNN speech enhancement systems typically estimate one or more masks (e.g., ideal ratio masks, spectral size masks, and / or the use of spectral features) that can be applied to the noisy speech spectrum. These masks selectively attenuate or amplify various frequency components, aiming at suppressing the noise while preserving the speech. However, the time-frequency based DNN speech enhancement systems ignore (fail to affect) the phase component due to its complexity. This results in less accurate estimation and / or improvements.The waveform domain-based DNN speech enhancement systems typically process and enhance raw speech waveform to output a pure speech waveform. However, waveform domain-based DNN speech enhancement systems are typically computationally complex due to their dependence on DNNs with large model sizes and their need for intensive training. This complexity presents challenges for real-time processing, particularly in devices with limited resources.Aspects and embodiments of the present disclosure address these and other limitations of existing technology by providing a machine learning (ML) model to a speech enhancement system. According to some implementations, the machine learning model includes an encoder, a bottleneck, and a decoder with multiple hop connections that each directly connect an encoder layer of the encoder to a decoder layer of the decoder. According to some embodiments, the number of encoder layers of the encoder is substantially equivalent to a number of decoder layers of the decoder. Each encoder layer of the encoder includes a residual network block (SE block), such as a Res2Net, and a compression excitation (SE) block. The residue network block contains a hierarchical residue-type connection that allows for multiple sizes of inclusion fields resulting in multiple feature scales. The SE block adaptively re-calibrates the channelwise feature responses. The bottleneck includes a sequence of layers of unidirectional controlled recurrent units (GRU layers) (e.g., a first GRU layer and a second GRU layer).In operation, the machine learning (ML) model receives a speech waveform (e.g., an original speech waveform or a first speech waveform). The original speech waveform is passed to the coder of the ML model. Each encoder layer of the encoder generates a latent output of the original speech waveform (e.g., feature maps). Specifically, for example, a residual mesh block of a respective encoder layer of the encoder receives feature maps (or latent outputs of the previous encoder layer) and generates multi-scale feature maps having multiple sizes of the recording fields. The SE block of the respective encoder layer of the encoder receives the multi-scale feature maps. The SE block compresses the multiscale feature maps into an average activation vector. The SE block generates a vector of the scale values using the average activation vector. The SE block re-calibrates the multi-scale feature maps using the vector of the scale values. The SE block combines the re-calibrated multi-scale feature maps with the original multi-scale feature maps to generate enhanced feature maps. The encoder layer creates a latent representation of the enhanced feature maps (e.g., a latent output). In addition, each encoder layer performs down sampling of the input while increasing the number of channels. The output of the last encoder layer of the encoder (e.g., a latent output) is passed to a bottleneck.The bottleneck receives the output of the last encoder layer of the encoder (e.g. the latent output) and outputs a non-linear transformation of the output of the last encoder layer of the encoder. The output of the bottleneck is forwarded to the decoder.Each decoder layer of the decoder increases the dimensionality of an input (e.g., the output of the bottleneck or an output of a previous decoder layer) by up-sampling and bisecting the number of channels. Each decoder layer further receives an output of the corresponding encoder layer via a hopping connection connected to a corresponding encoder layer. The output of the corresponding encoder layer containing detailed features and the output of the bottleneck containing the abstract features assist each decoder layer in reconstructing a speech waveform. In addition, each decoder layer performs up-sampling of the input while reducing the number of channels. Typically, the final decoder layer of the decoder results in a single channel which is used to reconstruct a speech waveform having the same dimension as the original speech waveform. The speech waveform reconstructed from the single channel represents a predicted speech waveform substantially equivalent to an enhanced speech waveform (or a second speech waveform) associated with the original speech waveform (or the first speech waveform).During training, the ML model is optimized using a loss function calculated based on (i) a difference between a waveform of the clean speech waveform of the training data associated with an original speech waveform of the training data and a waveform of an output of the ML model (e.g., an enhanced speech waveform) associated with the original speech waveform of the training data, and (ii) a difference between a size spectrogram of the clean speech waveform of the training data associated with the original speech waveform of the training data and a size spectrogram of the output of the ML model associated with the original speech waveform of the training data.Aspects of the present disclosure overcome these deficiencies and others by reducing computational complexity of the machine learning model used to predict an improved speech waveform from an original speech waveform while improving performance of the prediction.FIG. 1 illustrates an example speech enhancement system 100, in accordance with implementations of the present disclosure. The speech enhancement system 100 includes a machine learning (ML) model 150 of waveform enhancement (also referred to herein as an "ML model") that can receive an original speech waveform 110 and output an enhanced speech waveform 180. According to some embodiments, the original speech waveform 110 represented as x verrauscht may be pure speech represented as x, which may be corrupted by a convolutional space impulse response (convolutional RIR) (e.g., h) and / or additive background noise (e.g., x noise). Accordingly, the original speech waveform 110 (e.g., x verrauscht) may be mathematically expressed by: Equation (1).The ML model 150 operates as a denoiser function that predicts a speech waveform (e.g., x pred) by removing the noise (e.g., x noise) from an original speech waveform (e.g., x verrauscht). The predicted speech waveform (e.g., x pred) is substantially equivalent to an improved speech waveform (e.g., x). The ML model 150 can be mathematically expressed by: Equation (2).The ML model 150 may be deployed on an edge device to provide speech enhancement functionality. The edge device may be a computing device with moderate processing and storage capabilities that may have access (e.g., via an Internet of Things or IoT network) to local data and to a cloud service. The computing device may be implemented on (or shared by) any number of computing devices and / or implemented in a cloud. The computing device may be a desktop computer, a laptop computer, a smartphone, a tablet computer, a server, a computing device that accesses a remote server, a computing device that utilizes a virtualized computing environment, a gaming console, a portable computer, a smart TV, etc. The computing device may include any number of central processing units (CPUs) and graphics processing units (GPUs), including virtual CPUs and / or virtual GPUs, or other suitable processing devices. The computing device may further include any number of storage devices, network controllers, peripherals, and the like. The peripheral devices may include various scanning devices, photo cameras, video cameras, microphones, scanners, or any other device for capturing data.FIG. 2 is an example illustration of the architecture of the ML model 150 of FIG. 1, in accordance with implementations of the present disclosure. The ML model 150 may include an encoder 210, a bottleneck 220, and a decoder 230. The bottleneck 220 is arranged between the encoder 210 and the decoder 230. The ML model 150 may further include hop connections 240A-C. Each hop link of hop link 240A-C connects, in reverse order, an encoder layer of encoder 210 directly to a decoder layer of decoder 230. A first encoder layer of the encoder 210 is directly connected to a last decoder layer of the decoder 230, for example, via a first hop connection (e.g., the hop connection 240A).Referring briefly to FIG. 3A, encoder 210 receives original speech waveform 110 and processes original speech waveform 110 through multiple encoder layers (e.g., encoder layers 310A-C of FIG. 3B ) to generate a latent output. Each encoder layer can be mathematically expressed by equation (3), where E represents an encoder layer of encoder 210, x verrauscht represents the original speech waveform, and z represents a latent output. The first encoder layer of encoder 210 receives the original speech waveform (x verrauscht) and generates a latent output. z. Each subsequent encoder layer receives as an input the latent output of the previous encoder layer having 2 i-2 H channels and generates a latent output represented as min(2 i-1 H, Cm) channels, where H refers to the number of hidden channels of the first encoder layer, Cm refers to a maximum allowed channel dimension, and i refers to the ith encoder layer of the encoder layers of encoder 310A-C. In other words, each encoder layer performs down sampling of the input data from the previous encoder layer while increasing the number of channels.In FIG. 3B, each encoder layer (e.g., encoder layers 310A-C) includes a convolutional neural network (CNN) block 320, a residual network block 330, a compression and excitation (SE) block 340, and a CNN block 350.The CNN block 320 includes a stepped 1-D convolutional layer (e.g., conv1D layer), a rectified linear unit (ReLU) activation function, and a stack normalization (BN). The stepped conv1D layer refers to a set of more trainable filters (or kernels) that slide over one-dimensional input data based on a step. According to some embodiments, the stepped conv1D layer is indeterminate, indicating that the output at any particular point of the sequence depends only on the input. The step refers to the number of positions that the filter moves across the input data at each step. A step greater than 1 results in a down sampling which effectively reduces the length of the output sequence as compared to the input. In some embodiments, step (of 2) indicates that 2 downsamples are performed in each encoder layer.Each filter convolves over the input data to produce a transformed version of the data. The number of filters determines the number of generated feature maps, also known as output channels. In other words, each filter processes all input channels simultaneously and generates a single feature map (or output channel). Therefore, an increased number of filters directly results in an increased number of output channels (or feature maps). The stepped conv1D layer samples the length of the input down and increases the number of output channels. For example, each encoder layer inputs 2 i-2 H channels and outputs min (2 i-1 H, Cm) channels, where H refers to the number of hidden channels of the first encoder layer, Cm refers to a maximum allowed channel dimension, and i refers to the ith encoder layer of the one or more encoder layers 310A-C. In some embodiments, Cm may be set to a predetermined number (e.g., 768).The ReLU activation function, when applied to the feature maps (e.g., the output of the stepped 1-D convolution), introduces nonlinearity to improve the representation of important patterns and characteristics within the feature maps. The BN normalizes the feature maps by adjusting their mean and variance. The BN stabilizes and speeds up the learning process and ensures a consistent feature scale for more effective training of the subsequent layers.The residue network block 330, such as a Res2Net, receives an output of the CNN block 320 (e.g., feature maps). The residue mesh block 330 allows for the presence of multiple sizes of acquisition fields, resulting in the extraction of features on multiple scales. Referring again to FIG. 4, the residual mesh block 330 divides the feature maps equally into a predetermined number of subsets by the channel dimension, e.g., x 1, x 2, x 3, x 4. In other words, the predetermined number of subsets may be represented as s, each subset being represented as x i and i ∈{1,2,...,s}. Each subset other than the first subset (e.g., x 1) has a corresponding convolutional filter represented as K i( e.g., K 2, K 3, K 4). Each convolutional filter is a matrix of weights configured to extract the features over a scale range to increase the number of available capture fields for input. Each convolutional filter may be, for example, a 3×3 convolutional filter with a total of 9 weights (or trainable parameters) that may be adjusted during training.The residue network block 330 generates an output represented as y i for each subset. The output can be expressed mathematically by: equation (4) where i=1 with the residue net block 330 outputting x i as y i( i.e., y 1= x 1) where i=2, with the residue net block 330 processing x i by a convolutional filter K i and outputting y i( i.e., y 2= K 2( x 2)) and where i>2, wherein the residue net block 330 processes x i and y i-1 by a convolutional filter K i and outputs y i( i.e., y 3= K 3( x 3+ y 2)). As a result, early subsets (e.g., y 1, y 2) have smaller acquisition fields, capturing finer detail, while later subsets (e.g., y 3, y 4,...) effectively have larger acquisition fields due to the combination of inputs. Thus, the larger capture fields capture more comprehensive contextual information. Depending on the embodiment, the context information for the output of each subset (e.g., y 1, y 2, y 3, y 4) may be augmented using a combination of an augmented convolution with a predetermined augmentation factor (e.g., 2), a ReLU activation function, and a BN.The residue net block 330 may concatenate the outputs of the residue net block 330 (e.g., y 1, y 2, y 3, y 4) along the channel dimension. This results in new feature maps (e.g., feature maps y) with a spatial resolution that is the same as that of the input feature maps, but with a significantly higher number of channels. The feature maps y collect a more comprehensive range of information and allow the network to learn more complex features. The residue network block 330 forwards the feature maps y to the SE block 340.In FIG. 5, the SE block 340 (of FIG. 3 ) receives the feature maps y from the residue network block 330. The SE block 340 passes the feature maps y to the residue mesh block 330 to improve the feature map y (e.g., enhanced feature maps y') by assigning adaptive weights to each channel. SE block 340 includes a residual block 510, a compression block 520, an excitation block 530, a scaling block 540, and a residual link 550.The residual block 510 receives the feature maps y and extracts and / or transforms the features, resulting in modified feature maps y. The modified feature maps y are passed to the compression block 520. The compression block 520 compresses each channel into a single numerical value using the global average summary (GAP) layer. The GAP layer may be mathematically expressed by: equation (5) where zcrepresents an output value for the channel c of the merged feature maps (e.g., merging the modified feature maps y), y t represents a value at time t representing a row and column of the channel c in the modified feature maps y, and T represents the spatial dimensions (height and width) of the changed feature maps y. Accordingly, for each channel c of the modified feature maps y, the GAP layer iterates through all elements t, sums the values of all elements in the channel, dividing the sum by the total number of elements in the channel (T), the resulting value zcrepresents the average activation for the channel c in the merged feature maps. The zc across all channels together form a vector ž that captures a representation of the global information in the input feature map.The vector ž is passed to the excitation block 530, which calculates the weights for each channel using a first fully connected layer (FC layer), a nonlinear activation function, a second FC layer, and a sigmoid activation, resulting in a vector of the scale values s. The excitation block 530 may be mathematically expressed by: Equation (6).With respect to equation (6), the first FC layer characterized by the weight matrix W 1 reduces the dimensionality of the vector ž. The non-linear activation function, represented as f( ), is applied to the output of the first FC layer to introduce non-linearity. The second FC layer, characterized by the weight matrix W 1 receives the output of the nonlinear activation function and increases dimensionality back to its original size. Sigmoid activation, represented as σ, receives the output of the second FC layer and transforms each element of the output of the second FC layer into a value between 0 and 1. The vector of the scale values includes the channel weights used to re-calibrate the feature maps by scaling each channel based on the learned importance.The vector of the scale values (e.g., vector s) is passed to the scale block 540. In addition to the scaling block 540 receiving the vector s, the scaling block 540 receives the modified feature maps y. The scaling block 540 calibrates the modified feature maps y by performing channel-wise multiplication of the modified feature maps y with the vector of the scale values (e.g., the vector s). Thus, scaling block 540 adaptively emphasize or suppress various features in modified feature maps y. Scaling block 540 (or channel-by-channel multiplication) may be mathematically expressed by: equation (7), where y c is the c-th channel of modified feature maps y, s c is the c-th scale value of s, and z c is the re-calibrated c-th channel of modified feature maps y. In other words, the channel-by-channel multiplication is an element-by-element operation. For each channel in the modified feature maps y, each element of that channel is multiplied by a single scalar value (the channel specific weight). This operation scales the intensity of the features in each channel. The features of the higher weight channels are enhanced (or emphasized), while in those with lower weights, the features are attenuated (or suppressed). Accordingly, the output of the scaling block 540 is re-calibrated feature maps.The re-calibrated feature maps are passed to the residual link 550. In addition to the residual link 550 receiving the re-calibrated feature maps, the residual link 550 receives the feature maps y. The residual link 550, which is represented as a "+" sign, adds the feature maps y and the re-calibrated feature maps to generate the enhanced feature map (e.g., the enhanced feature maps y'). The enhanced feature map (e.g., the enhanced feature maps y') is an enriched feature representation that combines original and learned features. The SE block 340 passes the output of the SE block 340 (e.g., the enhanced feature maps y') to the CNN block 350.The CNN block 350 receives the enhanced feature maps y'. The CNN block 350 includes a convolutional layer (e.g., a 1×1 convolutional layer) and a controlled linear unit (GLU). The convolutional layer can be used to double the number of channels in the enhanced feature maps y'. The GLU can be used to bisect the number of times that are stored in the enhanced feature maps y' based on the number of filters. The GLU is an activation function that combines the outputs of two linear units using a control mechanism, thereby allowing the network to selectively focus on the relevant information. The output of the CNN block 350 is a latent representation of the enhanced feature maps y' (e.g., a latent output z). A latent representation refers to a learned abstract representation of the enhanced feature maps y' that captures substantial information while typically in a compressed or simplified form.In FIG. 3A, the initial encoder layer (e.g., encoder layer 310A) receives an original speech waveform 110 of FIG. 1. the initial encoder layer generates a latent output of the original speech waveform 110 (e.g., an initial latent output) from the original speech waveform 110. The initial latent output encapsulates certain aspects of the original speech waveform 110. The initial latent output is then fed to a subsequent encoder layer (e.g., encoder layer 310B). The subsequent encoder layer further processes the initial latent output to produce a latent output of the initial latent output (e.g., a subsequent latent output) that extracts potentially refined features or higher level features. This is accomplished, in accordance with some embodiments, by down-sampling a previous latent output (e.g., the initial latent output) in length and increasing the number of channels of the previous latent output. In other words, each subsequent encoder layer (e.g., encoder layer 310B and encoder layer 310C) receives a latent output of the previous encoder layer (e.g., encoder layer 310A and encoder layer 310B, respectively). The subsequent encoder layer processes the latent output of the previous encoder layer to produce a subsequent latent output that encapsulates certain aspects of the latent output of the previous encoder layer with more refined features or higher level features. This is repeated until the last encoder layer (e.g., encoder layer 310C) outputs a final latent output. The encoder 210 passes the final latent output z to the bottleneck 220.The bottleneck 220 receives the final latent output z and outputs a non-linear transformation of the final latent output z (e.g., a latent output ž). According to some embodiments, the final latent output z is the same size as the latent output ž. In FIG. 6, bottleneck 220 is a sequence modelling network including a first controlled recurrent unit (GRU) layer 610A and a second GRU layer 610B. Each of the GRU layers (e.g., the GRU layer 610A and the GRU layer 610B) is a unidirectional GRU, which is a type of recurrent neural network (RNN) used for processing sequential data in a direction from the beginning to the end of the sequence. Each of the GRU layers may include hidden units, also referred to as individual neurons positioned between the input and output layers. According to some embodiments, the number of hidden units min(2 L-1 H, Cm) may be hidden units, where L refers to the number of encoder layers in encoder 210.Each of the GRU layers may be mathematically expressed by: equation (8), where a GRU refers to a GRU layer (e.g., GRU layer 610A and GRU layer 610B), and ž is the non-linear transformation of the input latent output z. Accordingly, the GRU layer 610A receives the input latent output z received from the encoder 210 and generates a non-linear transform of the latent output z (e.g., the latent output ž). The GRU layer 610A forwards the latent output ž to the GRU layer 610B. The GRU layer 610B receives the input latent output z (e.g., the latent output ž received from the GRU layer 610A) and generates a non-linear transformation of the input latent output (e.g., the latent output ž). In other words, each GRU layer (e.g., GRU layer 610A and GRU layer 610B) performs a non-linear transformation on an input. It should be noted that other RNNs may be considered, where one or more of the GRU layers may be replaced with a long short-term memory (LSTM), a bidirectional LSTM with multiple self-consideration layers, etc. The latent output ž is passed to the decoder 230.Decoder 230 generates a predicted speech waveform (e.g., x pred), associated with the original speech waveform (e.g., x verrauscht) by processing latent output ž through multiple decoder layers. According to some embodiments, the number of decoder layers of decoder 230 corresponds to the number of encoder layers of encoder 210. Decoder 230 may be mathematically expressed by equation (9), where D refers to decoder 230 that includes multiple decoder layers, ž is the latent output received from bottleneck 220, and x̂ corresponds to enhanced speech waveform 180. Each decoder layer of decoder 230 includes a convolution layer (e.g., a 1×1 convolution layer), a controlled linear unit (GLU), and a transposed 1D convolution. The dio Di DiConvolutional layer can be used to double the number of channels. The GLU can be used to divide the number of channels by half. Transposed 1D convolution, also known as a fractional-stepped convolution or deconvolution, is one type of convolution operation used to increase dimensionality by up-sampling and bisecting the number of channels. Additionally, as stated above, the hop connection connecting a respective decoder layer to a respective encoder layer of the encoder 210 provides an output of the respective encoder layer to the respective decoder. Thus, the latent output from the corresponding encoder layer (i.e., the detailed features) and the latent output (i.e., abstract features) received from the encoder 210 assist the decoder in reconstructing the speech waveform. Typically, the last layer of decoder 230 results in a single channel that is used to reconstruct a speech waveform having the same dimension as the original speech waveform.For example, the initial decoder layer receives the latent output from encoder 210. In addition, the initial decoder layer receives the latent output from a corresponding encoder layer of decoder 210. The initial decoder layer reconstructs a speech waveform. This is accomplished by up-sampling and decreasing the number of channels, in accordance with some embodiments. The speech waveform is then fed to a subsequent decoder layer. The subsequent decoder layer further reconstructs a subsequent speech waveform. As stated above, this can be accomplished by up-sampling and reducing the number of channels. The subsequent decoder layer uses the speech waveform received from the initial decoder layer and a latent output from a corresponding encoder layer of the encoder 210 to generate the subsequent speech waveform. In other words, each subsequent decoder layer receives a speech waveform from the previous decoder layer and the latent output from a corresponding encoder layer. The subsequent decoder layer reconstructs a subsequent speech waveform. This is repeated until the last decoder layer, which outputs a speech waveform reconstructed from a single channel having the same dimension as the original speech waveform 110 (e.g., x verrauscht). The speech waveform reconstructed from a single channel represents a predicted speech waveform (e.g., x pred), which is associated with the original speech waveform 110 (e.g., x verrauscht). As noted above, the predicted speech waveform (e.g., x pred) is substantially equivalent to the enhanced speech waveform 180 (e.g., x̂).FIG. 7 is an exemplary illustration of a system 700 for training the ML model 150 in accordance with implementations of the present disclosure. The system 700 includes a dataset 710, an ML model 150, and a loss function 780.According to one embodiment, dataset 710 includes a collection of clean speech records from different speakers (e.g., ~40 speakers) having different accents and speech styles, and a collection of noisy speech records obtained by adding different types of noise (e.g., gemurmel, auto noise, street noise) at different signal to noise ratios (SNRs). Training set 720 contains multiple utterances (e.g., ~ 12,000) from multiple speakers (e.g., ~30 speakers). Each utterance of training set 720 includes clean utterances mixed with background sounds having different types of sounds taken from dataset 710 (e.g., ~8 different types of sounds) and artificial types of sounds at different SNRs (e.g., 4 different SNRs). The test set 730 includes multiple utterances (e.g., ~1,000) from multiple unknown speakers (e.g., ∼2). Each utterance of the test set 730 is mixed with background sounds of different types of sounds taken from the dataset 710 (e.g., ~5 different types of sounds) and at different SNRs (e.g., 4 different SNRs). According to some embodiments, the different SNRs used for training set 720 are different from the different SNRs used for test set 730. According to some embodiments, the different SNRs used for training set 720 are the same as the different SNRs used for test set 730.According to another embodiment, dataset 710 includes a collection of audio clips from multiple speakers (e.g., ~10,000 speakers) at a predetermined sampling frequency (e.g., 16 kHz). The clean amount contains a predetermined amount of clean speech (e.g., ~500 hours of clean red). The amount of noise includes a predetermined amount of noise (e.g., ~180 hours) from different noise categories (e.g., 150 noise classes). Training set 720 includes multiple clean-noise pairs (e.g., 500 hours clean and noisy speech pairs) at different SNR levels (e.g., 31 SNR levels between -5 and 25 dB) with a predetermined silence duration (e.g., 0.2 seconds) for each clean speech waveform. The test set 730 includes multiple artificial (or synthetic) clean-noise pairs (e.g., 150 pairs) with and without Hall.During training of the ML model 150, each training data pair (x verrauscht, x) of the training set 720 is randomly cut into a clip of predetermined length (e.g., a T-second clip). Data augmentation such as mixing the sounds within a stack, removing a fraction of the frequency evenly, adding decaying echoes of clean speech, and sounds may be applied to the cut clip. All training data of the training set 720 is forwarded to the ML model 150 in order to output an improved speech waveform x̂. The enhanced speech waveform x̂ is passed to loss function 780. A pure speech waveform x of the test set 730 is passed to the loss function 780. Loss function 780 receives enhanced speech waveform x̂ and clean speech waveform x.The loss function 780 is used to optimize the ML model 150. Loss function 780 includes L1 distance loss over the waveform and Multi-Resolution Short Time Fourier Transform (MRSTFT) loss over the size spectrogram. Both losses are calculated between the pure speech waveform x and the enhanced speech waveform x̂. The MRSTFT loss is defined as a sum of the spectral convergence loss (sc loss) and the magnitude loss (mag loss). The MRSTFT loss, referred to as L MRSTFT may be mathematically expressed by equation (10), where S is (x; φ i) = | STFT(x)|, and refers to the linear scale spectrogram of x, S(x̂; φ i) = | STFT(x̂)| and refers to the linear scale spectrogram of x̂, φ i representing STFT hyperparameters at the ith resolution, | | | | F and | | | | 1 refer to the Frobenius and L1 norms, respectively, and M refers to the total number of resolutions. The STFT hyperparameters include, for example, a number of FFT classes, skip sizes, window lengths, etc. The number of FFT classes may be, for example, ∈{512, 1024, 848}. The jump sizes may be, for example, ∈{50, 120, 240}. The window lengths may be, for example, ∈{240, 600, 1200}.Given MRSTFT loss, the resulting loss function 780 may be mathematically expressed by equation (11). The loss function 780 is used for model optimization. Specifically, the result of loss function 780, represented as L (e.g., result 790), is fed back to ML model 150, computing a gradient with respect to each parameter. Additional operations, such as the Adam Optimizer with pulse, may be performed on the gradient to further optimize the ML model. The pulse adam optimizer, also known as the AMSGrad, adaptively adjusts the learning rate for each parameter based on its individual gradients and the prior squared gradients and accumulates the prior gradients to build up a pulse in the direction of decay. Additionally, during training, linear warm-up with cosine cool learning rate may be used. Loss function 780 in conjunction with other optimization techniques simply adjusts the internal parameters (weights and distortions) of ML model 150 to minimize overall loss. The processes of prediction by the ML model 150, loss calculation by the loss function 780, and parameter settings are repeated through different iterations over the training data.FIG. 8 illustrates a flow diagram of an example method 800 for processing a waveform using the speech enhancement system, in accordance with implementations of the present disclosure. The method 800 may be performed by processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. According to one implementation, some or all of the operations of method 800 may be performed by ML model 150 of FIG. 1, as described above.At block 810, the processing logic receives an original speech waveform through the speech enhancement system. As described above, the original speech waveform is clean speech, which may be corrupted by convolutional RIR and / or additive background noise. The speech enhancement system functions as a denoiser function predicting speech waveform substantially equivalent to enhanced speech waveform.At block 820, the processing logic generates a latent output of the original speech waveform. As described above, an encoder of the speech enhancement system is used to generate the latent output of the original speech waveform. In particular, each encoder layer of the encoder includes a residue network block and an SE block. The residual mesh block of a respective encoder layer receives the feature maps (or the latent output of the previous encoder layer) and generates multi-scale feature maps having multiple magnitudes of the recording fields.The SE block of a respective encoder layer receives the multi-scale feature maps from the residue network block. The SE block compresses the multiscale feature maps into an average activation vector. The SE block generates a vector of the scale values using the average activation vector. The SE block re-calibrates the multi-scale feature maps using the vector of the scale values. The SE block combines the re-calibrated multi-scale feature maps with the original multi-scale feature maps to generate enhanced feature maps. The encoder layer creates a latent representation of the enhanced feature maps (e.g., a latent output). The output of the last encoder layer represents the latent output of the original speech waveform.According to some embodiments, a bottleneck of the speech enhancement system receives the latent output of the original speech waveform, and outputs a non-linear transform of the latent output of the original speech waveform, which may be passed to the decoder.At block 830, the processing logic reconstructs an enhanced speech waveform associated with the original speech waveform from the latent output. The latent output refers to the latent output of the original speech waveform. As described above, each decoder layer of the decoder increases the dimensionality of an input (e.g., the latent outputs or an output of a previous decoder layer) by up-sampling and bisecting the number of channels. Each decoder layer further receives an output of the corresponding encoder layer via a hopping connection connected to a corresponding encoder layer. The output of the corresponding encoder layer containing detailed features and the latent output containing abstract features assist each decoder layer in reconstructing a speech waveform. In addition, each decoder layer performs up-sampling of the input while reducing the number of channels. The output of the final decoder layer of the decoder is a speech waveform reconstructed from a single channel (e.g., an enhanced speech waveform). According to one embodiment, the speech waveform output by the last decoder layer has the same dimension as the original speech waveform.FIG. 9 is a block diagram illustrating an example computer system 900 in accordance with implementations of the present disclosure. The computer system 900 may correspond to an edge device in which the ML model 150 described with reference to FIG. 1 is deployed. The computer system 900 may operate in the capacity of a server or an endpoint machine in an endpoint server network environment, or as a peer machine in a peer-to-peer network environment (or a distributed network environment). The machine may be a television, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a server, a network router, a network switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying the operations to be performed by that machine. Further, while only a single machine is illustrated, the term "machine" is also intended to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform one or more of the methodologies discussed herein.The example computer system 900 includes a processing device (processor) 902, a main memory 904 (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), double data rate memory (DDR SDRAM), or DRAM (RDRAM), etc.), a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 918 that communicate with each other via a bus 940.The processor (processing device) 902 represents one or more general purpose processing devices, such as a microprocessor, a central processing unit, or the like. In particular, processor 902 may be a complex instruction set (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processor 902 may also be one or more special-purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. Processor 902 may include processing logic 922 used to perform the operations discussed herein. The processor 902 is configured to execute the instructions 905 to perform the operations discussed herein.The computer system 900 may further include a network interface device 908. The computer system 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 912 (e.g., a keyboard, an alphanumeric keyboard, a motion scan input device, a touchscreen), a cursor control device 914 (e.g., a mouse), and a signal generation device 920 (e.g., a speaker).The data storage device 918 may include a non-transitory machine-readable storage medium 924 (also a computer-readable storage medium) storing one or more sets of instructions 926 embodying any one or more of the methodologies or functions described herein. The instructions may also reside, completely or at least partially, within the main memory 904 and / or within the processor 902 during execution thereof by the computer system 900, wherein the main memory 904 and the processor 902 also form machine-readable storage media. The instructions may further be transmitted or received over a network 930 via the network interface device 908.While the computer readable storage medium 924 (the machine readable storage medium) is shown as a single medium, according to an example implementation, the terms "computer readable storage medium" and "machine readable storage medium" should be construed to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store the one or more sets of instructions. The terms "computer-readable storage medium" and "machine-readable storage medium" are also intended to include any medium that can store, encode, or transmit a set of instructions for execution by the machine that cause the machine to perform any one or more of the methodologies of the present disclosure. The terms "computer readable storage medium" and "machine readable storage medium" are accordingly intended to be construed to include, but are not limited to, solid state memories, optical media, and magnetic media.Reference throughout this specification to "a single implementation", "a single embodiment", "an implementation", or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the implementation and / or embodiment is included in at least one implementation and / or embodiment. Thus, the occurrence of the phrase "according to a single implementation" or "according to an implementation" at various places throughout this specification may refer to the same implementation, but not necessarily, depending on circumstances. Further, the particular features, structures, or characteristics may be combined in any suitable manner, according to one or more implementations.To the extent that the terms "comprises," "comprising," "includes," "including," "has," "includes," variants thereof, and other similar words are used in either the detailed description or the claims, these terms are intended to be inclusive as an open-ended transition word in a manner similar to the term "comprising," without excluding any additional or other elements.As used in this application, the terms "block", "layer", "component", "module", "system", or the like are generally intended to refer to a computer-related entity, either hardware (e.g., circuitry), software, a combination of hardware and software, or an entity that relates to an operating machine having one or more specific functionalities. A component may be, for example, but is not limited to, a process executing on a processor (e.g., a digital signal processor), a processor, an object, an executable, an execution thread, a program, and / or a computer. For purposes of illustration, both an application executing on a controller and the controller may be a component. One or more components may be located within a process and / or an execution tree, where a component is located on a computer and / or distributed between two or more computers. A "device" may also occur in the form of specially designed hardware; generalized hardware specialized by execution of software thereon, which allows the hardware to perform specific functions (e.g., generate points of interest and / or descriptors); software in a computer readable medium; or a combination thereof.The above-mentioned systems, circuits, modules, etc. have been described with respect to the interaction between multiple components and / or blocks. It can be appreciated that such systems, circuits, components, blocks, etc. may include these components or specified subcomponents, some of the specified components or subcomponents, and / or additional components according to various permutations and combinations of the foregoing. The subcomponents may also be implemented as components communicatively coupled to other components rather than being included within predecessor components (hierarchically). Additionally, it should be noted that one or more components may be combined into a single component that provides overall functionality or may be split into multiple separate subcomponents, and that one or more middle layers, such as a management layer, may be provided to be communicatively coupled to such subcomponents to provide integrated functionality. All components described herein may also cooperate with one or more other components not specifically described herein, but known to those skilled in the art.Moreover, the words "example" or "exemplary" are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, the use of the words "example" or "example" is intended to represent the concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or.". That is, unless otherwise specified or clear from context, "X uses A or B" is intended to mean any of the natural inclusive permutations. That is, when X uses A, X uses B, or X uses both A and B, then "X uses A or B" is satisfied in each of the foregoing cases. In addition, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless otherwise specified or clear from context to be directed to a singular form.Finally, the implementations described herein include a collection of data describing a user and / or the activities of a user. According to one implementation, such data is only collected when the user provides his consent to collect that data. According to some implementations, the user is prompted to expressly permit data collection. Further, the user may agree or deny approval to participate in such data collection activities. According to one implementation, the collected data is anonymous before an analysis is performed to obtain statistical patterns so that the identity of the user cannot be determined from the collected data.

Claims

A method comprising: - receiving a first speech waveform; - processing an input based on the first speech waveform using a trained machine learning model to generate a second speech waveform, the trained machine learning model comprising: - an encoder, the encoder comprising a plurality of encoder layers, each encoder layer comprising: - receiving, by a residue network block of a respective encoder layer, as input one of: a first speech waveform or multi-scale feature maps output from a previous encoder layer of the plurality of encoder layers; - generating, based on the input, multi-scale feature maps of the respective encoder layer.The method of claim 1, wherein the residue network block is a Res2Net.The method according to any of the preceding claims, wherein each encoder layer further comprises: - receiving an output of the residue network block of the respective encoder layer through a compression and excitation, SE, block of the respective encoder layer; - adaptively re-calibrating the multi-scale feature maps based on the received output of the residue network block of the respective encoder layer.The method of claim 3, wherein each encoder layer further comprises: - receiving an output of the SE block of the respective encoder layer by an extended convolution of the respective encoder layer; - enlarging, by the extended convolution of the respective encoder layer, the acquisition fields of the multi-scale feature maps without loss of spatial resolution; - receiving, by a rectified linear unit, ReLU, activation function, activation function of the respective encoder layer, an output of the extended convolution of the respective encoder layer; - introducing a non-linearity by the ReLU activation function of the respective encoder layer; - receiving an output of the ReLU activation function of the respective encoder layer by a stack normalization, BN, of the respective encoder layer; normalizing an output of the ReLU activation function of the respective encoder layer by the stack normalization of the respective encoder layer.The method according to any of the preceding claims, wherein the trained machine learning model further comprises: - a bottleneck arranged between the encoder and a decoder of the machine learning model, the bottleneck comprising: - receiving an output of the encoder through a first layer of controlled recurrent units, GRU, layer; - executing through the first GRU layer a first non-linear transformation; - receiving an output of the first GRU layer through a second GRU layer; - executing through the second GRU layer a second non-linear transformation.The method of claim 5, wherein the first GRU layer and the second GRU layer are a unidirectional GRU layer.The method according to any of the preceding claims, wherein the trained machine learning model further comprises: - a decoder, wherein the decoder comprises a decoder of the encoder layers; - a plurality of hop connections, wherein each hop connection connects a decoder layer of the decoder layers to an encoder layer of the encoder.The method of any preceding claim, wherein the first speech waveform is substantially equivalent to a second speech waveform including at least one of: a convolution space impulse response, convolution RIR, and additive background noise.A non-transitory computer readable medium comprising instructions that, in response to execution by a processing device, cause the processing device to perform operations comprising: - receiving a plurality of speech waveform pairs, each speech waveform pair of the plurality of speech waveform pairs including a first speech waveform and a second speech waveform; - training a machine learning model to generate a third speech waveform based on the first speech waveform of a respective speech waveform, wherein training the machine learning model comprises: - inputting the first speech waveform of the respective speech waveform into the machine learning model; - predicting the third speech waveform based on the inputted first speech waveform; - inputting the second speech waveform of the respective speech waveform and the third speech waveform into a loss function; optimizing the machine learning model based on a result of the loss function, wherein the result of the loss function is calculated based on (i) a difference between the second speech waveform of the respective speech waveform and the third speech waveform and (ii) a difference between a magnitude spectrogram of the second speech waveform of the respective speech waveform and a magnitude spectrogram of the third speech waveform.The non-transitory computer readable medium of claim 9, wherein the first speech waveform of the respective speech waveform is the second speech waveform of the respective speech waveform, comprising at least one of: a convolution space impulse response, convolution RIR, and additive background noise.The non-transitory computer readable medium of any of claims 9 or 10, wherein the machine learning model comprises: - an encoder, wherein the encoder comprises multiple encoder layers, each encoder layer comprising: - a residue network block, wherein the residue network block receives one of: the first speech waveform of the respective speech waveform or multi-scale feature maps output from a previous encoder layer of the multiple encoder layers to generate multi-scale feature maps of a respective encoder layer.The non-transitory computer readable medium of claim 11, wherein the residue network block is a Res2Net.The non-transitory computer readable medium of any of claims 11 or 12, wherein each encoder layer further comprises: - a compression and excitation, SE, block, wherein the SE block receives an output of the residue network block of the respective encoder layer and adaptively recalibrations the multi-scale feature maps.The non-transitory computer readable medium of claim 13, wherein each encoder layer further comprises: an extended convolution, the extended convolution receiving an output signal of the SE block of the respective encoder layer and increasing the acquisition fields of the multi-scale feature maps without losing spatial resolution; an Rectified Linear Unit, ReLU, activation function, the ReLU activation function receiving the output of the extended convolution of the respective encoder layer and introducing a nonlinearity; a Stack Normalization, BN, wherein the stack normalization receives the output of the ReLU activation function of the respective encoder layer and normalizes the output of the ReLU activation function of the respective encoder layer.The non-transitory computer readable medium of any one of claims 9 to 14, wherein the machine learning model further comprises: a bottleneck disposed between an encoder and a decoder of the machine learning model, the bottleneck comprising: a first controlled recurrent units, GRU, layer, the first GRU layer receiving an output of the encoder and executing a first non-linear transformation; a second GRU layer, the second GRU layer receiving an output of the first GRU layer and executing a second non-linear transformation.A system comprising: a storage device; and a processing device coupled to the storage device, the processing device to perform operations comprising: receiving, by a speech enhancement system, a first speech waveform; generating, by an encoder of the speech enhancement system, a latent output of the first speech waveform, wherein each layer of a plurality of encoder layers of the encoder comprises a residue network block and a compression and excitation, SE, block; performing, by a bottleneck of the speech enhancement system, a nonlinear transformation on the latent output of the first speech waveform, the bottleneck comprising a first layer of controlled recurrent units, GRU, layer, and a second GRU layer; predicting, by a decoder of the speech enhancement system, a second speech waveform from the non-linear transformation of the latent output of the first speech waveform.The system of claim 16, wherein the first speech waveform is substantially equivalent to a second speech waveform including at least one of: a convolution space impulse response, convolution RIR, and additive background noise.The system of any of claims 16 or 17, wherein generating the latent output of the first speech waveform comprises: - for each encoder layer of the plurality of encoder layers, generating, by the residue network block of a respective encoder layer, multi-scale feature maps associated with an input of the respective encoder layer, - adaptively recalibration of the multi-scale feature maps by the SE block of the respective encoder layer, wherein an output of the respective encoder layer is a latent output of the input of the respective encoder layer.The system of any of claims 16 to 18, wherein performing the nonlinear transformation on the latent output of the first speech waveform by the bottleneck of the speech enhancement system comprises: - receiving the latent output of the first speech waveform; - performing, by the first GRU layer, a first nonlinear transformation on the latent output of the first speech waveform; - receiving an output of the first GRU layer by the second GRU layer; - performing, by the second GRU layer, a second nonlinear transformation.The system according to any of claims 16 to 19, wherein predicting the second speech waveform from the non-linear transformation of the latent output of the first speech waveform comprises: - for each decoder layer of a plurality of decoder layers of the decoder, receiving a latent output of an encoder layer of the plurality of encoder layers connected to a respective decoder layer via a hopping connection.

Images