Machine learning model architecture for speech enhancement system

Abstract

Methods and systems for a machine learning model architecture for a speech enhancement system. The disclosed machine learning model architecture comprises an encoder, a decoder, a bottleneck arranged between the encoder and the decoder and the like. The encoder includes a plurality of encoder layers, and the decoder includes a plurality of decoder layers. Each encoder layer is connected to a corresponding decoder layer via a hopping connection. Each encoder layer includes a Res2Net and a compression and excitation (SE) block. The bottleneck includes a first gated recursive unit (GRU) layer and a second GRU layer.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 536,034, filed on August 31, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various aspects and implementations of the present disclosure relate to a machine learning model architecture for a speech enhancement system. Background Art

[0004] Speech enhancement systems have become integral to improving the perceived quality and intelligibility of speech waveforms contaminated by additive background noise and reverberation. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The various aspects and implementations of the present disclosure will be more fully understood from the specific embodiments given below and from the accompanying drawings of various aspects and implementations of the present disclosure, however, these aspects and implementations should not be considered to limit the present disclosure to specific aspects or implementations, but are only for explanation and understanding.

[0006] Figure 1 An example speech enhancement system according to implementations of the present disclosure is shown.

[0007] Figure 2 is an exemplary illustration of a machine learning model architecture for speech enhancement according to an implementation of the present disclosure.

[0008] Figure 3A and 3B According to the implementation of the present disclosure Figure 2 An exemplary illustration of an encoder for a machine learning model.

[0009] Figure 4 is an exemplary illustration of a Res2Net layer of the encoder layer of the encoder of FIG. 3 according to an implementation of the present disclosure.

[0010] Figure 5 is an exemplary illustration of squeeze-and-excitation (SE) layers of encoder layers of the encoder of FIG. 3 according to an implementation of the present disclosure.

[0011] Figure 6 According to the implementation of the present disclosure Figure 2 An exemplary illustration of the bottleneck of a machine learning model.

[0012] Figure 7 is a training according to the implementation of the present disclosure Figure 2 An exemplary illustration of a machine learning model.

[0013] Figure 8 A flow chart of an example method for processing a waveform using a speech enhancement system according to implementations of the present disclosure is depicted.

[0014] Fig. 9 is a block diagram illustrating an exemplary computer system according to an implementation of the present disclosure. DETAILED DESCRIPTION

[0015] Various aspects of the present disclosure relate to machine learning model architectures for speech enhancement systems. Speech enhancement systems are typically designed to improve the quality and intelligibility of speech signals, particularly in environments with noise or other forms of interference. Speech enhancement systems are of vital importance in a variety of applications, for example, in speech recognition, hearing aids, and modern telecommunication systems.

[0016] Conventional speech enhancement systems utilize one or more methods to achieve better noise reduction and speech quality. For example, some speech enhancement systems (such as speech enhancement systems using spectral subtraction combined with Wiener filtering) generate an estimate of the distribution or composition of frequencies present in the noise (e.g., noise spectrum) of the speech signal and / or an adjustment of the amplitude (e.g., gain mask) of the speech signal. Therefore, the speech enhancement system multiplies the estimated gain mask of the speech signal with the noise spectrum to produce an enhanced spectrum. Conventional speech enhancement systems operate under the assumption that the noise is stationary or slowly changing, such as constant background hum, fan noise, and slowly changing traffic noise. In real-world environments, noise is usually non-stationary; it changes rapidly over time. Non-stationary noise includes, for example, background conversations, slamming doors, car honking, dog barking, bird chirping, keyboard typing, etc. Therefore, conventional speech enhancement systems tend to fail in reducing noise and improving speech quality. This is further complicated under low signal-to-noise ratio (SNR) conditions (wherein the noise level is high compared to the speech level), making it difficult to distinguish between noise and speech, resulting in less accurate estimates.

[0017] In recent years, there has been an increasing use of machine learning and deep learning algorithms as they are better suited to handle non-stationary noise and maintain speech quality. Most notably, deep neural networks (DNNs) have been implemented in speech enhancement systems (e.g., DNN speech enhancement systems). DNN speech enhancement systems have significantly advanced the field, providing more robust performance than traditional speech enhancement systems under a variety of noise conditions, including non-stationary noise and low SNR conditions. DNN speech enhancement systems are typically time-frequency based or waveform-based.

[0018] Time-frequency based DNN speech enhancement systems typically estimate one or more masks (e.g., ideal ratio masks, spectral amplitude masks, and / or use of spectral features) that can be applied to the noisy speech spectrum. These masks selectively attenuate or amplify different frequency components, aiming to suppress noise while preserving speech. However, time-frequency based DNN speech enhancement systems ignore (fail to manipulate) the phase component due to their complexity. This results in less accurate estimates and / or enhancements.

[0019] Waveform-domain based DNN speech enhancement systems usually process and enhance the original speech waveform to output a clean speech waveform. However, waveform-domain based DNN speech enhancement systems are usually computationally complex due to their reliance on DNNs with large model sizes and the need for intensive training. This complexity poses challenges for real-time processing, especially for resource-constrained devices.

[0020] Aspects and embodiments of the present disclosure address these and other limitations of the prior art by providing a machine learning (ML) model for a speech enhancement system. In some implementations, the machine learning model includes an encoder, a bottleneck, and a decoder with multiple skip connections, each of which connects the encoder layer of the encoder directly to the decoder layer of the decoder. In some embodiments, the number of encoder layers of the encoder is substantially equivalent to the number of decoder layers of the decoder. Each encoder layer of the encoder includes a residual network block (such as Res2Net) and a compression-excitation (SE) block. The residual network block includes a hierarchical class residual connection that implements a plurality of receptive fields of size, resulting in a plurality of feature scalings. The SE block adaptively recalibrates the channel-by-channel feature response. The bottleneck includes a sequence of unidirectional gated recurrent unit (GRU) layers (e.g., a first GRU layer and a second GRU layer).

[0021] During operation, a machine learning (ML) model receives a speech waveform (e.g., an original speech waveform or a first speech waveform). The original speech waveform is forwarded to an encoder of the ML model. Each encoder layer of the encoder generates a potential output (e.g., a feature map) of the original speech waveform. Specifically, for example, a residual network block of a corresponding encoder layer of the encoder receives a feature map (or a potential output of a previous encoder layer) and generates a multi-scale feature map with a receptive field of multiple sizes. The SE block of the corresponding encoder layer of the encoder receives the multi-scale feature map. The SE block compresses the multi-scale feature map to an average activation vector. The SE block uses the average activation vector to generate a scaling value vector. The SE block uses the scaling value vector to recalibrate the multi-scale feature map. The SE block combines the recalibrated multi-scale feature map with the original multi-scale feature map to generate an enhanced feature map. The encoder layer generates a potential representation of the enhanced feature map (e.g., potential output). In addition, each encoder layer downsamples the input while increasing the number of channels. The output (e.g., potential output) of the last encoder layer of the encoder is forwarded to the bottleneck.

[0022] The bottleneck receives the output of the last encoder layer of the encoder (eg, the potential output) and outputs a non-linear transformation of the output of the last encoder layer of the encoder. The output of the bottleneck is forwarded to the decoder.

[0023] Each decoder layer of the decoder increases the dimension of the input (e.g., the output of the bottleneck or the output of the previous decoder layer) by upsampling and halving the number of channels. Each decoder layer also receives the output of the corresponding encoder layer via a jump connection connected to the corresponding encoder layer. The output of the corresponding encoder layer including detailed features and the output of the bottleneck including abstract features assist each decoder layer in reconstructing the speech waveform. In addition, each decoder layer upsamples the input while reducing the number of channels. Typically, the last decoder layer of the decoder generates a single channel for reconstructing a speech waveform with the same dimension as the original speech waveform. The speech waveform reconstructed from a single channel represents a predicted speech waveform, which is substantially equivalent to the enhanced speech waveform (or second speech waveform) associated with the original speech waveform (or the first speech waveform).

[0024] During training, the ML model is optimized using a loss function calculated based on: (i) the difference between a waveform of a clean speech waveform of the training data associated with an original speech waveform of the training data and a waveform of an output of the ML model associated with the original speech waveform of the training data (e.g., an enhanced speech waveform), and (ii) the difference between an amplitude spectrogram of a clean speech waveform of the training data associated with an original speech waveform of the training data and an amplitude spectrogram of the output of the ML model associated with the original speech waveform of the training data.

[0025] Aspects of the present disclosure overcome these and other deficiencies by reducing the computational complexity of a machine learning model used to predict an enhanced speech waveform from an original speech waveform while improving the performance of the prediction.

[0026] Figure 1 An example speech enhancement system 100 according to an implementation of the present disclosure is shown. The speech enhancement system 100 includes a waveform enhancement machine learning (ML) model 150 (also referred to herein as an “ML model”) that can receive an original speech waveform 110 and output an enhanced speech waveform 180. In some embodiments, the original speech waveform 110 (denoted as x noisy ) can be composed of clean speech (denoted as x), which can be convolved with a room impulse response (RIR) (e.g., h) and / or additive background noise (e.g., x noise ) is destroyed. Therefore, mathematically, the original speech waveform 110 (e.g., x noisy ) can be expressed by equation (1): noisy =x*h+x noise .

[0027] The ML model 150 is constructed by extracting the original speech waveform (e.g., x noisy ) to remove noise (e.g., x noise ) as a denoiser function for predicting the speech waveform (e.g., x pred ) to operate. pred ) is substantially equivalent to the enhanced speech waveform (e.g., x). Mathematically, the ML model 150 can be expressed by equation (2): pred =f(x noisy )≈x.

[0028] The ML model 150 can be deployed to edge devices to provide speech enhancement capabilities. Edge devices can be computing devices with moderate processing and memory capabilities, and can access local data (e.g., via an Internet of Things or IoT network) and cloud services. Computing devices can be implemented on any number of computing devices and / or on the cloud (or shared between them). Computing devices can be desktop computers, laptop computers, smart phones, tablet computers, servers, computing devices that access remote servers, computing devices that utilize virtualized computing environments, game consoles, wearable computers, smart TVs, etc. The computing device can have any number of central processing units (CPUs) and graphics processing units (GPUs), including virtual CPUs and / or virtual GPUs or any other suitable processing devices. The computing device can also have any number of memory devices, network controllers, peripheral devices, etc. Peripheral devices can include various sensing devices, photographic cameras, video cameras, microphones, scanners, or any other devices for data ingestion.

[0029] Figure 2 According to the implementation of the present disclosure Figure 1 1 is an exemplary illustration of the architecture of the ML model 150 of FIG. 1. The ML model 150 may include an encoder 210, a bottleneck 220, and a decoder 230. The bottleneck 220 is disposed between the encoder 210 and the decoder 230. The ML model 150 may also include skip connections 240A-C. Each of the skip connections 240A-C directly connects an encoder layer of the encoder 210 to a decoder layer of the decoder 230 in reverse order. For example, the first encoder layer of the encoder 210 is directly connected to the last decoder layer of the decoder 230 via a first skip connection (e.g., skip connection 240A).

[0030] Quick Reference Figure 3A , the encoder 210 receives the original speech waveform 110 and passes it through multiple encoder layers (e.g., Figure 3B The encoder layers 310A-C) process the original speech waveform 110 to generate potential outputs. Mathematically, each encoder layer can be expressed by the following formula: z = E (x noisy ), where E represents the encoder layer of encoder 210, x noisy represents the original speech waveform, and z represents the potential output. The first encoder layer of the encoder 210 receives the original speech waveform (x noisy ) and generates a potential output z. Each subsequent encoder layer receives a i-2 The latent output of the previous encoder layer with H channels is taken as input and the generated code is represented as min(2 i-1H, Cm) channels, where H refers to the number of hidden channels of the first encoder layer, Cm refers to the maximum allowed channel dimension, and i refers to the i-th encoder layer of the encoder layers of encoders 310A-C. In other words, each encoder layer downsamples the data from the input of the previous encoder layer while increasing the number of channels.

[0031] refer to Figure 3B , each encoder layer (e.g., encoder layers 310A-C) includes a convolutional neural network (CNN) block 320, a residual network block 330, a squeeze and excitation (SE) block 340, and a CNN block 350.

[0032] CNN block 320 includes a strided 1-D convolution (e.g., conv1D) layer, a rectified linear unit (ReLU) activation function, and batch normalization (BN). A strided conv1D layer refers to a set of learnable filters (or kernels) that slide on one-dimensional input data based on a stride. In some embodiments, the strided conv1D layer is arbitrary, indicating that the output at any given point in the sequence depends only on the input. The stride refers to the number of positions that the filter moves on the input data in each step. A stride greater than 1 results in downsampling, which effectively reduces the length of the output sequence compared to the input. In some embodiments, the stride (which is 2) indicates that it is downsampled by 2 in each encoder layer.

[0033] Each filter convolves over the input data to produce a transformed version of the data. The number of filters determines the number of feature maps (also known as output channels) generated. In other words, each filter processes all input channels simultaneously, producing a single feature map (or output channel). Therefore, increasing the number of filters directly leads to an increased number of output channels (or feature maps). The strided conv1D layer downsamples the input in length and increases the number of output channels. For example, each encoder layer inputs 2 i-2 H channels and output min(2 i-1 H, Cm) channels, where H refers to the number of hidden channels of the first encoder layer, Cm refers to the maximum allowed channel dimension, and i refers to the i-th encoder layer of the one or more encoder layers 310A-C. In some embodiments, Cm can be set to a predetermined number (e.g., 768).

[0034] When applied to a feature map (e.g., the output of a strided 1-D convolution), the ReLU activation function introduces nonlinearity to enhance the representation of important patterns and features within the feature map. BN normalizes the feature map by adjusting its mean and variance. BN effectively stabilizes and accelerates the learning process, and ensures consistent feature scaling for more efficient training of subsequent layers.

[0035] The residual network block 330 (such as Res2Net) receives the output (e.g., feature map) of the CNN block 320. The residual network block 330 enables the presence of receptive fields of multiple sizes, resulting in features being extracted at multiple scales. Quick Reference Figure 4 , the residual network block 330 evenly divides the feature map into a predetermined number of subsets according to the channel dimension, for example, x1, x2, x3, x4. In other words, the predetermined number of subsets can be represented as s, where each subset is represented as x i , and where i∈{1,2,···,s}. Each subset excluding the first subset (e.g., x1) has a corresponding convolution filter, denoted as K i (e.g., K2, K3, K4). Each convolution filter is a weight matrix that is configured to extract features across a zoom range to increase the number of available receptive fields for the input. For example, each convolution filter can be a 3x3 convolution filter with a total of 9 weights (or learnable parameters) that can be adjusted during training.

[0036] The residual network block 330 generates an output for each subset, denoted as y i Mathematically, the output can be expressed by equation (4): Where i = 1, the residual network block 330 outputs x i As y i (i.e., y1=x1), where i=2, the residual network block 330 passes through the convolution filter K i Processing x i , and output y i (i.e., y2=K2(x2)), and where i>2, the residual network block 330 passes through the convolution filter K i Processing x i and i-1 , and output y i (ie, y3 = K3 (x 3+ y2)). As a result, the early subsets (e.g., y1, y2) have smaller receptive fields, thereby capturing finer details, while the later subsets (e.g., y3, y4, etc.) effectively have larger receptive fields due to the combination of the inputs. Therefore, the larger receptive fields capture a wider range of contextual information. Depending on the embodiment, a combination of dilated convolutions with a predetermined dilation factor (e.g., 2), a ReLU activation function, and BN can be used to increase the contextual information of the outputs (e.g., y1, y2, y3, y4) for each subset.

[0037] The residual network block 330 can cascade the outputs of the residual network block 330 (e.g., y1, y2, y3, y4) along the channel dimension. This results in a new feature map (e.g., feature map y) with the same spatial resolution as the input feature map but with a significantly larger number of channels. The feature map y captures a wider range of information and allows the network to learn more complex features. The residual network block 330 forwards the feature map y to the SE block 340.

[0038] refer to Figure 5 , the SE block 340 (of FIG. 3 ) receives the feature map y from the residual network block 330. The SE block 340 passes the feature map y to the residual network block 330 to enhance the feature map y (e.g., enhanced feature map y′) by assigning adaptive weights to each channel. The SE block 340 includes a residual block 510, a compression block 520, an excitation block 530, a scaling block 540, and a residual connection 550.

[0039] The residual block 510 receives the feature map y and extracts and / or transforms features, thereby generating a modified feature map y. The modified feature map y is forwarded to the compression block 520. The compression block 520 compresses each channel to a single value using a global average pooling (GAP) layer. Mathematically, the GAP layer can be expressed by equation (5): Among them, z C represents the output value of channel c of the pooled feature map (e.g., the pooling of the modified feature map y), y t represents the value at t, t represents the row and column of channel c in the modified feature map y, and T represents the spatial dimensions (height and width) of the modified feature map y. Therefore, for each channel c of the modified feature map y, the GAP layer traverses all elements t, the GAP layer sums the values ​​of all elements in the channel, divides the sum by the total number of elements in the channel (T), and the resulting value z C represents the average activation of channel c in the pooled feature map. z across all channels C Together they form a vector representation that captures the global information in the input feature map

[0040] vector is forwarded to the excitation block 530, which uses a first fully connected (FC) layer, a nonlinear activation function, a second FC layer, and a sigmoid activation to calculate the weight of each channel, thereby producing a vector of scaled values ​​s. Mathematically, the excitation block 530 can be expressed by equation (6):

[0041] Given equation (6), the first FC layer represented by the weight matrix W1 reduces the vector dimensionality. A nonlinear activation function denoted as f() is applied to the output of the first FC layer to introduce nonlinearity. The second FC layer represented by the weight matrix W1 receives the output of the nonlinear activation function and increases the dimensionality back to its original size. The sigmoid activation denoted as σ receives the output of the second FC layer and transforms each element of the output of the second FC layer to a value between 0 and 1. The output of the sigmoid activation denoted as s as described above refers to a vector of scaling values, one scaling value for each channel. The vector of scaling values ​​contains the channel weights used to adaptively recalibrate the feature map by scaling each channel based on learned importance.

[0042] The vector of scaling values ​​(e.g., vector s) is forwarded to the scaling block 540. In addition to receiving the vector s, the scaling block 540 also receives the modified feature map y. The scaling block 540 recalibrates the modified feature map y by performing a channel-by-channel multiplication on the modified feature map y and the vector of scaling values ​​(e.g., vector s). Thus, the scaling block 540 adaptively emphasizes or suppresses different features in the modified feature map y. Mathematically, the scaling block 540 (or the channel-by-channel multiplication) can be expressed by equation (7): z c =s c *y c , where y c is the cth channel of the modified feature map y, s c is the cth scaled value of s, z c is the recalibrated cth channel of the modified feature map y. In other words, the channel-by-channel multiplication is an element-by-element operation. For each channel in the modified feature map y, each element of the channel is multiplied by a single scalar value (channel-specific weight). This operation scales the intensity of the features in each channel. Channels with higher weights will have their features amplified (or emphasized), while channels with lower weights will have their features attenuated (or suppressed). Therefore, the output of the scaling block 540 is a recalibrated feature map.

[0043] The recalibrated feature map is forwarded to the residual connection 550. In addition to receiving the recalibrated feature map, the residual connection 550 also receives the feature map y. The residual connection 550, represented as a "+" symbol, adds the feature map y to the recalibrated feature map to generate an enhanced feature map (e.g., enhanced feature map y'). The enhanced feature map (e.g., enhanced feature map y') is a rich feature representation that combines the original features and the learned features. The SE block 340 forwards the output of the SE block 340 (e.g., enhanced feature map y') to the CNN block 350.

[0044] The CNN block 350 receives the enhanced feature map y'. The CNN block 350 includes a convolutional layer (e.g., a 1x1 convolutional layer) and a gated linear unit (GLU). The convolutional layer can be used to double the number of channels in the enhanced feature map y'. The GLU can be used to halve the number of channels in the enhanced feature map y' based on the number of filters. The GLU is an activation function that combines the outputs of two linear units using a gating mechanism, allowing the network to selectively focus on relevant information. The output of the CNN block 350 is a potential representation of the enhanced feature map y' (e.g., potential output z). The potential representation refers to a learned abstract representation of the enhanced feature map y' that captures essential information while typically having a more compressed or simplified form.

[0045] refer to Figure 3A , the initial encoder layer (e.g., encoder layer 310A) receives Figure 1 The original speech waveform 110. The initial encoder layer generates the potential output (e.g., initial potential output) of the original speech waveform 110 from the original speech waveform 110. The initial potential output encapsulates certain aspects of the original speech waveform 110. The initial potential output is then fed to a subsequent encoder layer (e.g., encoder layer 310B). The subsequent encoder layer further processes the initial potential output to generate a potential output (e.g., subsequent potential output) of the initial potential output, which potentially extracts more refined or higher-level features. In some embodiments, this is achieved by downsampling the previous potential output (e.g., initial potential output) in length and increasing the number of channels of the previous potential output. In other words, each subsequent encoder layer (e.g., encoder layer 310B and encoder layer 310C) receives the potential output of the previous encoder layer (e.g., encoder layer 310A and encoder layer 310B, respectively). The subsequent encoder layer processes the potential output of the previous encoder layer to generate a subsequent potential output, which encapsulates certain aspects of the potential output of the previous encoder layer with more refined or higher-level features. This operation is repeated until the last encoder layer (eg, encoder layer 310C) outputs the final potential output. Encoder 210 forwards the final potential output z to bottleneck 220.

[0046] Bottleneck 220 receives the final potential output z and outputs a nonlinear transformation of the final potential output z (e.g., potential output ). In some embodiments, the final potential output z is equal to the potential output The same size as Figure 6, bottleneck 220 is a sequence modeling network that includes a first gated recurrent unit (GRU) layer 610A, a second GRU layer 610B. Each of the GRU layers (e.g., GRU layer 610A and GRU layer 610B) is a unidirectional GRU for processing sequential data in one direction from the beginning to the end of the sequence, which is a type of recurrent neural network (RNN). Each of the GRU layers may include hidden units (also called individual neurons) located between the input layer and the output layer. In some embodiments, the number of hidden units may be min(2 L-1 H, Cm) hidden units, where L refers to the number of encoder layers in encoder 210.

[0047] Mathematically, each of the GRU layers can be expressed by equation (8): Wherein, GRU refers to a GRU layer (e.g., GRU layer 610A and GRU layer 610B), and is a nonlinear transformation of the potential output z of the input. Thus, the GRU layer 610A receives the potential output z of the input received from the encoder 210 and generates a nonlinear transformation of the potential output z (e.g., the potential output ). GRU layer 610A converts the potential output Forwarded to GRU layer 610B. GRU layer 610B receives the input potential output z (e.g., the potential output z received from GRU layer 610A) ), and generates a nonlinear transformation of the potential output of the input (e.g., the potential output ). In other words, each GRU layer (e.g., GRU layer 610A and GRU layer 610B) performs a nonlinear transformation on the input. It should be noted that other RNNs can be considered, and one or more of the GRU layers can be replaced by a long short-term memory (LSTM), a bidirectional LSTM with multiple self-attention layers, etc. Potential output is forwarded to the decoder 230 .

[0048] Decoder 230 processes the potential outputs by passing them through a number of decoder layers. to generate the same waveform as the original speech (e.g., x noisy ) is associated with a predicted speech waveform (e.g., x pred ). In some embodiments, the number of decoder layers of decoder 230 matches the number of encoder layers of encoder 210. Mathematically, decoder 230 can be expressed by the following formula:

[0049] Equation (9):

[0050] Wherein D refers to a decoder 230 including a plurality of decoder layers, is the potential output received from the bottleneck 220, and x^ corresponds to the enhanced speech waveform 180. Each decoder layer of the decoder 230 includes a convolutional layer (e.g., a 1x1 convolutional layer), a gated linear unit (GLU), and a transposed 1D convolution. The convolutional layer can be used to double the number of channels. The GLU can be used to halve the number of channels. The transposed 1D convolution (also referred to as fractional stride convolution or deconvolution) is a type of convolution operation for increasing the dimension by upsampling and halving the number of channels. In addition, as described above, the jump connection connecting the corresponding decoder layer to the corresponding encoder layer of the encoder 210 provides the output of the corresponding encoder layer to the corresponding decoder. Therefore, the potential output (i.e., detailed features) from the corresponding encoder layer and the potential output (i.e., abstract features) received from the encoder 210 assist the decoder layer in reconstructing the speech waveform. Typically, the last decoder layer of the decoder 230 generates a single channel for reconstructing a speech waveform with the same dimension as the original speech waveform.

[0051] For example, the initial decoder layer receives potential outputs from the encoder 210. In addition, the initial decoder layer receives potential outputs from the corresponding encoder layer of the encoder 210. The initial decoder layer reconstructs the speech waveform. In some embodiments, this is achieved by upsampling and reducing the number of channels. The speech waveform is then fed to the subsequent decoder layer. The subsequent decoder layer further reconstructs the subsequent speech waveform. As described above, this can be achieved by upsampling and reducing the number of channels. The subsequent decoder layer uses the speech waveform received from the initial decoder layer and the potential outputs from the corresponding encoder layer of the encoder 210 to generate the subsequent speech waveform. In other words, each subsequent decoder layer receives the speech waveform from the previous decoder layer and receives the potential outputs from the corresponding encoder layer. The subsequent decoder layer reconstructs the subsequent speech waveform. Repeat this operation until the output is from a layer having a signal similar to the original speech waveform 110 (e.g., x noisy ) is used to reconstruct a speech waveform of a single channel of the same dimension as the original speech waveform 110 (e.g., x). noisy ) is associated with a predicted speech waveform (e.g., x pred As described above, the predicted speech waveform (e.g., x pred ) is substantially equivalent to the enhanced speech waveform 180 (e.g., ).

[0052] Figure 7 7 is an exemplary illustration of a system 700 for training an ML model 150 according to an implementation of the present disclosure. The system 700 includes a dataset 710, an ML model 150, and a loss function 780.

[0053] In one embodiment, the data set 710 includes a collection of clean speech recordings with different accents and speaking styles from various speakers (e.g., about 40 speakers) and a collection of noisy speech recordings obtained by adding various types of noise (e.g., noisy human voices, car noise, street noise) at different signal-to-noise ratios (SNRs). The training set 720 includes multiple utterances (e.g., about 12,000) from multiple speakers (e.g., about 30 speakers). Each utterance of the training set 720 includes a clean utterance mixed with background noise, and the background noise has various different noise types (e.g., about 8 different noise types) taken from the data set 710 and artificial noise types at various SNRs (e.g., 4 different SNRs). The test set 730 includes multiple utterances (e.g., about 1,000) from multiple unseen speakers (e.g., about 2). Each utterance of the test set 730 is mixed with background noise having various different noise types (e.g., about 5 different noise types) and various SNRs (e.g., 4 different SNRs) taken from the dataset 710. In some embodiments, the various SNRs used for the training set 720 are different from the various SNRs used for the test set 730. In some embodiments, the various SNRs used for the training set 720 are the same as the various SNRs used for the test set 730.

[0054] In another embodiment, the data set 710 includes a collection of audio clips with a predetermined sampling frequency (e.g., 16kHz) from multiple speakers (e.g., about 10,000 speakers). The clean set includes a predetermined amount of clean speech (e.g., about 500 hours of clean speech). The noise set includes a predetermined amount of noise (e.g., about 180 hours) from various noise categories (e.g., 150 noise categories). The training set 720 includes multiple clean-noise pairs (e.g., 500 hours of clean speech and noise speech pairs) at various SNR levels (e.g., 31 SNR levels between -5dB and 25dB) with a predetermined silence length (e.g., 2 seconds) for each clean speech waveform. The test set 730 includes multiple artificial (or synthetic) clean-noise pairs (e.g., 150 pairs) with and without reverberation.

[0055] During the training of the ML model 150, each training data pair (x noisy ,x) are randomly cropped into clips of a predetermined length (e.g., T second clips). Data enhancement can be applied to the cropped clips, such as shuffling noise within a batch, uniformly removing a portion of frequencies, adding decaying echoes to clean speech and noise. Each training data of the training set 720 is forwarded to the ML model 150 to output an enhanced speech waveform Enhanced speech waveform is forwarded to the loss function 780. The clean speech waveform x of the test set 730 is forwarded to the loss function 780. The loss function 780 receives the enhanced speech waveform and clean speech waveform x.

[0056] The loss function 780 is used for optimization of the ML model 150. The loss function 780 incorporates the L1 distance loss on the waveform and the multi-resolution short-time Fourier transform (MRSTFT) loss on the magnitude spectrogram. The two losses are calculated between. The MRSTFT loss is defined as the sum of the spectral convergence (sc) loss and the amplitude (mag) loss. Mathematically, it is called L MRSTFT The MRSTFT loss can be expressed by equation (10): Among them, S(x; φ i ) = |STFT(x)| and refers to the linearly scaled amplitude spectrum of x, refers to the linearly scaled amplitude spectrum of x^, φ i represents the STFT hyperparameters at the i-th resolution, |||| F and ||||1 refer to the Frobenius and L1 norms, respectively, and M refers to the total number of resolutions. STFT hyperparameters include, for example, the number of FFT bins, the hop size, the window length, etc. The number of FFT bins can be, for example, ∈{512, 1024, 2048}. The hop size can be, for example, ∈{50, 120, 240}. The window length can be, for example, ∈{240, 600, 1200}.

[0057] Given the MRSTFT loss, mathematically, the resulting loss function 780 can be expressed by equation (11): The loss function 780 is used for model optimization. In particular, the result of the loss function 780, represented as L (e.g., result 790), is fed back into the ML model 150, and the gradients for each parameter are calculated. Additional operations (such as an Adam optimizer with momentum) can be performed on the gradients to further optimize the ML model. The Adam optimizer with momentum (also known as AMSGrad) adaptively adjusts the learning rate of each parameter based on its individual gradient and past squared gradients, and accumulates past gradients to build momentum in the direction of descent. In addition, a linear ramp with a cosine annealing learning rate can be used during training. Simply, the loss function 780 adjusts the internal parameters (weights and biases) of the ML model 150 in combination with other optimization methods to minimize the overall loss. The process of prediction of the ML model 150, loss calculation of the loss function 780, and parameter adjustment is repeated through various iterations of the training data.

[0058] Figure 8 A flow chart depicts an example method 800 for processing a waveform using a speech enhancement system according to an implementation of the present disclosure. The method 800 may be performed by processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. In one implementation, some or all of the operations of the method 800 may be executed by Figure 1 The ML model 150 is executed as described above.

[0059] At block 810, processing logic receives an original speech waveform via a speech enhancement system. As previously described, the original speech waveform consists of clean speech that may be corrupted by convolutional RIR and / or additive background noise. The speech enhancement system operates as a denoiser function that predicts a speech waveform that is substantially equivalent to the enhanced speech waveform.

[0060] At block 820, processing logic generates a potential output of the original speech waveform. As previously described, the encoder of the speech enhancement system is used to generate a potential output of the original speech waveform. Specifically, each encoder layer of the encoder includes a residual network block and an SE block. The residual network block of the corresponding encoder layer receives the feature map (or the potential output of the previous encoder layer) and generates a multi-scale feature map with a receptive field of multiple sizes.

[0061] The SE block of the corresponding encoder layer receives the multi-scale feature map from the residual network block. The SE block compresses the multi-scale feature map to an average activation vector. The SE block generates a scaling value vector using the average activation vector. The SE block uses the scaling value vector to recalibrate the multi-scale feature map. The SE block combines the recalibrated multi-scale feature map with the original multi-scale feature map to generate an enhanced feature map. The encoder layer generates a potential representation (e.g., potential output) of the enhanced feature map. The output of the last encoder layer represents the potential output of the original speech waveform.

[0062] In some embodiments, a bottleneck of the speech enhancement system receives a potential output of an original speech waveform and outputs a non-linear transformation of the potential output of the original speech waveform, which can be forwarded to a decoder.

[0063] At box 830, processing logic reconstructs an enhanced speech waveform associated with the original speech waveform from potential output. Potential output refers to the potential output of the original speech waveform. As previously described, each decoder layer of the decoder increases the dimensionality of the input (e.g., potential output or output of the previous decoder layer) by up-sampling and halving the number of channels. Each decoder layer also receives the output of the corresponding encoder layer via a jump connection connected to the corresponding encoder layer. The output of the corresponding encoder layer including detailed features and the potential output including abstract features assist each decoder layer in reconstructing the speech waveform. In addition, each decoder layer up-samples the input while reducing the number of channels. The output of the last decoder layer of the decoder is a speech waveform (e.g., enhanced speech waveform) reconstructed from a single channel. In some embodiments, the speech waveform output by the last decoder layer has the same dimension as the original speech waveform.

[0064] Fig. 9 900 is a block diagram showing an exemplary computer system 900 according to an implementation of the present disclosure. The computer system 900 may correspond to a computer system in which the Figure 1 The computer system 900 may be an edge device for the ML model 150 described. The computer system 900 may operate in the capacity of a server or endpoint machine in an endpoint-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a television, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a web appliance, a server, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine. In addition, while only a single machine is shown, the term "machine" should also be taken to include any collection of machines that individually or jointly execute one (or more) sets of instructions to perform any one or more of the methods discussed herein.

[0065] The example computer system 900 includes a processing device (processor) 902, a volatile memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (such as synchronous DRAM (SDRAM), double data rate (DDRSDRAM) or DRAM (RDRAM)), etc.), a non-volatile memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 918, which communicate with each other via a bus 940.

[0066] Processor (processing device) 902 represents one or more general processing devices, such as a microprocessor, a central processing unit, etc. More specifically, processor 902 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. Processor 902 may also be one or more special processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. Processor 902 may include processing logic 922 for performing the operations discussed herein. Processor 902 is configured to execute instructions 905 for performing the operations discussed herein.

[0067] The computer system 900 may also include a network interface device 908. The computer system 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), input devices 912 (e.g., a keyboard and alphanumeric keypad, motion sensing input devices, touch screens), a cursor control device 914 (e.g., a mouse), and signal generating devices 920 (e.g., speakers).

[0068] The data storage device 918 may include a non-transitory machine-readable storage medium 924 (also referred to as a computer-readable storage medium) on which is stored one or more sets of instructions 926 embodying any one or more of the methods or functions described herein. The instructions may also reside, completely or at least partially, within the volatile memory 904 and / or within the processor 902 during execution by the computer system 900, the volatile memory 904 and the processor 902 also constituting machine-readable storage media. Instructions may also be sent or received over the network 930 via the network interface device 908.

[0069] Although the computer-readable storage medium 924 (machine-readable storage medium) is shown as a single medium in the exemplary implementation, the terms "computer-readable storage medium" and "machine-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The terms "computer-readable storage medium" and "machine-readable storage medium" should also be considered to include any medium that can store, encode, or carry a set of instructions for execution by a machine and cause the machine to perform any one or more of the methods of the present disclosure. Therefore, the terms "computer-readable storage medium" and "machine-readable storage medium" should be considered to include, but are not limited to, solid-state memories, optical media, and magnetic media.

[0070] References throughout this specification to "one implementation," "one embodiment," "implementation," or "embodiment" mean that a particular feature, structure, or characteristic described in connection with the implementation and / or embodiment is included in at least one implementation and / or embodiment. Thus, appearances of the phrase "in one implementation" or "in an implementation" throughout this specification may, but do not necessarily, refer to the same implementation, depending on the circumstances. Furthermore, in one or more implementations, the particular features, structures, or characteristics may be combined in any suitable manner.

[0071] To the extent that the terms "comprise," "comprising," "includes," "including," "having," "containing," variations thereof, and other similar words are used in the detailed description or claims, these terms are intended to be inclusive in a manner similar to the term "comprising" as an open transition word without excluding any additional or other elements.

[0072] As used in this application, the terms "block", "layer", "component", "module", "system", etc. are generally intended to refer to a computer-related entity, either hardware (e.g., circuitry), software, a combination of hardware and software, or an entity related to an operating machine having one or more specific functions. For example, a component can be, but is not limited to, a process running on a processor (e.g., a digital signal processor), a processor, an object, an executable file, a thread of execution, a program, and / or a computer. As an illustration, both an application running on a controller and the controller can be components. One or more components can reside within a process and / or thread of execution, and a component can be localized on one computer and / or distributed between two or more computers. In addition, a "device" can take the form of specially designed hardware; general-purpose hardware that is specifically implemented by executing software thereon, which enables the hardware to perform specific functions (e.g., generating points of interest and / or descriptors); software on a computer-readable medium; or a combination thereof.

[0073] The aforementioned systems, circuits, modules, etc. have been described with respect to the interaction between several components and / or blocks. It is understood that such systems, circuits, components, blocks, etc. may include those components or specified subcomponents, some of the specified components or subcomponents, and / or additional components, and according to various permutations and combinations of the aforementioned. Subcomponents may also be implemented as components that are communicatively coupled to other components rather than being included in a parent component (layered). In addition, it should be noted that one or more components may be combined into a single component that provides aggregate functionality or divided into several separate subcomponents, and any one or more intermediate layers (such as a management layer) may be provided to communicatively couple to such subcomponents in order to provide integrated functionality. Any component described herein may also interact with one or more other components that are not specifically described herein but are known to those skilled in the art.

[0074] In addition, the words "example" or "exemplary" are used herein to mean used as an example, instance or illustration. Any aspect or design described herein as "exemplary" is not necessarily interpreted as being preferred over or having advantages over other aspects or designs. On the contrary, the use of the words "example" or "exemplary" is intended to present concepts in a specific way. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clearly known from the context, "X adopts A or B" is intended to mean any natural inclusive replacement. That is, if X adopts A; X adopts B; or X adopts both A and B, then "X adopts A or B" is satisfied under any of the aforementioned examples. In addition, unless otherwise specified or clearly known from the context to involve a singular form, the articles "a" and "an" as used in this application and the appended claims should generally be interpreted as meaning "one or more".

[0075] Finally, the implementations described herein include the collection of data describing a user and / or a user's activities. In one implementation, such data is collected only when the user provides consent to the collection of the data. In some implementations, the user is prompted to explicitly allow data collection. In addition, the user can choose to opt in or out of participating in such data collection activities. In one implementation, the collected data is anonymized before performing any analysis to obtain any statistical patterns, so that the identity of the user cannot be determined from the collected data.

Claims

1. A method comprising: receiving a first speech waveform; Processing an input based on the first speech waveform using a trained machine learning model to generate a second speech waveform, wherein the trained machine learning model includes: An encoder, wherein the encoder comprises a plurality of encoder layers, each encoder layer comprising: Receiving, by the residual network block of the corresponding encoder layer, one of the following as input: a first speech waveform or a multi-scale feature map output from a previous encoder layer among the plurality of encoder layers; and A multi-scale feature map of the corresponding encoder layer is generated based on the input.

2. The method according to claim 1, wherein: The residual network block is Res2Net.

3. The method according to claim 1, wherein: Each encoder layer also consists of: receiving, by a compression and excitation (SE) block of the corresponding encoder layer, an output of the residual network block of the corresponding encoder layer; and The multi-scale feature map is adaptively recalibrated based on the received output of the residual network block of the corresponding encoder layer.

4. The method according to claim 1, wherein: Each encoder layer also includes: receiving an output of the SE block of the corresponding encoder layer through a dilated convolution of the corresponding encoder layer; Increasing the receptive field of a multi-scale feature map by the dilated convolution of the corresponding encoder layer without losing spatial resolution; receiving an output of the dilated convolution of the corresponding encoder layer through a rectified linear unit (ReLU) activation function of the corresponding encoder layer; Introducing non-linearity via the ReLU activation function of the corresponding encoder layer; receiving an output of the ReLU activation function of the corresponding encoder layer through batch normalization (BN) of the corresponding encoder layer; and An output of the ReLU activation function of the corresponding encoder layer is normalized by the BN of the corresponding encoder layer.

5. The method according to claim 1, wherein: The trained machine learning model also includes: A bottleneck is set between the encoder and the decoder of the machine learning model, wherein the bottleneck comprises: receiving an output of the encoder through a first gated recurrent unit (GRU) layer; Performing a first nonlinear transformation by the first GRU layer; receiving an output of the first GRU layer through a second GRU layer; and A second nonlinear transformation is performed by the second GRU layer.

6. The method according to claim 5, wherein: The first GRU layer and the second GRU layer are unidirectional GRU layers.

7. The method according to claim 1, wherein: The trained machine learning model also includes: A decoder, wherein the decoder comprises a decoder of an encoder layer; and A plurality of skip connections, wherein each skip connection connects a decoder layer in the decoder layer to an encoder layer in the encoder.

8. The method according to claim 1, wherein: The first speech waveform is substantially identical to a second speech waveform comprising at least one of: a convolved room impulse response (RIR) and additive background noise.

9. A non-transitory computer-readable medium comprising instructions that, in response to being executed by a processing device, cause the processing device to perform operations comprising: Receiving a plurality of speech waveform pairs, wherein each of the plurality of speech waveform pairs comprises a first speech waveform and a second speech waveform; Training a machine learning model to generate a third speech waveform based on a first speech waveform in the corresponding speech waveforms, wherein training the machine learning model comprises: inputting the first speech waveform among the corresponding speech waveforms into the machine learning model; Predicting the third speech waveform based on the input first speech waveform; Inputting the second speech waveform and the third speech waveform in the corresponding speech waveform into the loss function; The machine learning model is optimized based on a result of the loss function, wherein the result of the loss function is calculated based on (i) a difference between the second speech waveform and the third speech waveform in the corresponding speech waveform, and (ii) a difference between an amplitude spectrogram of the second speech waveform and an amplitude spectrogram of the third speech waveform in the corresponding speech waveform.

10. The non-transitory computer readable medium of claim 9, wherein: The first one of the corresponding speech waveforms is the second one of the corresponding speech waveforms, the second speech waveform including at least one of the following: a convolution room impulse response (RIR) and additive background noise.

11. The non-transitory computer readable medium of claim 9, wherein: The machine learning model includes: An encoder, wherein the encoder comprises a plurality of encoder layers, each encoder layer comprising: A residual network block, wherein the residual network block receives one of the following items to generate a multi-scale feature map of a corresponding encoder layer: the first speech waveform in the corresponding speech waveform or a multi-scale feature map output from a previous encoder layer in the multiple encoder layers.

12. The non-transitory computer readable medium of claim 11, wherein: The residual network block is Res2Net.

13. The non-transitory computer readable medium of claim 11, wherein: Each encoder layer also includes: A squeeze and excitation (SE) block, wherein the SE block receives an output of the residual network block of the corresponding encoder layer and adaptively recalibrates the multi-scale feature map.

14. The non-transitory computer readable medium of claim 11, wherein: Each encoder layer also includes: A dilated convolution, wherein the dilated convolution receives the output of the SE block of the corresponding encoder layer and increases the receptive field of a multi-scale feature map without losing spatial resolution; a rectified linear unit (ReLU) activation function, wherein the ReLU activation function receives the output of the dilated convolution of the corresponding encoder layer and introduces nonlinearity; and Batch Normalization (BN), wherein the BN receives the output of the ReLU activation function of the corresponding encoder layer and normalizes the output of the ReLU activation function of the corresponding encoder layer.

15. The non-transitory computer readable medium of claim 9, wherein: The machine learning model also includes: A bottleneck is set between an encoder and a decoder of the machine learning model, wherein the bottleneck comprises: a first gated recurrent unit (GRU) layer, wherein the GRU layer receives an output of the encoder and performs a first non-linear transformation; and A second GRU layer, wherein the second GRU layer receives the output of the first GRU layer and performs a second non-linear transformation.

16. A system comprising: Memory devices; as well as a processing device coupled to the memory device, wherein the processing device is to perform operations comprising: receiving, by a speech enhancement system, a first speech waveform; generating a potential output of the first speech waveform by an encoder of the speech enhancement system, wherein each of a plurality of encoder layers of the encoder comprises a residual network block and a compression and excitation (SE) block; performing a non-linear transformation on the potential output of the first speech waveform through a bottleneck of the speech enhancement system, wherein the bottleneck comprises a first gated recurrent unit (GRU) layer and a second GRU layer; and A second speech waveform is predicted by a decoder of the speech enhancement system from the non-linear transformation of the potential output of the first speech waveform.

17. The system of claim 16, wherein: The first speech waveform is substantially identical to a second speech waveform comprising at least one of: a convolved room impulse response (RIR) and additive background noise.

18. The system of claim 16, wherein: Generating the potential output of the first speech waveform includes: For each encoder layer of the plurality of encoder layers, generating a multi-scale feature map associated with an input of the corresponding encoder layer by the residual network block of the corresponding encoder layer; and The multi-scale feature map is adaptively recalibrated by the SE block of the corresponding encoder layer, wherein the output of the corresponding encoder layer is a potential output of the input of the corresponding encoder layer.

19. The system of claim 16, wherein: Performing a non-linear transformation on the potential output of the first speech waveform by the bottleneck of the speech enhancement system comprises: receiving a potential output of the first speech waveform; performing a first nonlinear transformation on the potential output of the first speech waveform via the first GRU layer; receiving an output of the first GRU layer through the second GRU layer; and A second nonlinear transformation is performed by the second GRU layer.

20. The system of claim 16, wherein: Predicting the second speech waveform from the non-linear transformation of the potential output of the first speech waveform comprises: For each decoder layer of a plurality of decoder layers of the decoder, a potential output of an encoder layer of the plurality of encoder layers connected to the corresponding decoder layer via a skip connection is received.

Images