Deep Learning-Based Audio Artifact Reduction

Abstract

The system is programmed to construct a machine learning model including a sequence of masking blocks. Each masking block receives a feature vector of an audio segment. Each masking block includes a first component that generates a first mask for extracting clean speech and a second component that generates a second mask for extracting residual speech masked by the first mask. Each masking block also generates a specific feature vector based on the first mask and the second mask, which becomes the specific feature vector for the next masking block. The second component, which may include a layer of gated recurrent units, has lower computational complexity than the first component, which may include multiple convolutional layers. The system is further programmed to receive an input feature vector of an input audio segment and execute the machine learning model to obtain an output feature vector of an output audio segment.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to PCT Application No. PCT / CN2022 / 110612, filed August 5, 2022, U.S. Provisional Application No. 63 / 424,620, filed November 11, 2022, and European Patent Application No. 22214817.3, filed December 20, 2022, each of which is incorporated by reference in its entirety.

[0002] This application relates to audio processing and machine learning. [Background technology]

[0003] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived and pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0004] In recent years, various machine learning models have been adopted for speech enhancement. Compared with traditional signal processing methods such as Wiener filtering and spectral subtraction, machine learning methods have shown significant improvements, especially under non-stationary noise and low signal-to-noise ratio (SNR) conditions.

[0005] Existing machine learning methods for speech detection and enhancement often suffer from over-suppression of speech, which can even lead to speech distortion or choppiness. Furthermore, existing machine learning methods for speech detection and enhancement are each typically developed to mitigate one type of artifact, such as noise, reverberation echo, codec effects, packet loss, or feedback effects.

[0006] Improving traditional machine learning methods for speech enhancement should help to efficiently reduce speech over-suppression and mitigate multiple types of artifacts, especially in stored audio content and real-time communications. Summary of the Invention [Means for solving the problem]

[0007] A computer-implemented method for reducing audio artifacts is disclosed. The method includes receiving, by a processor, audio data as a joint time-frequency representation across multiple frames and multiple frequency bands. The method further includes executing, by the processor, a digital model for detecting speech from a feature vector of the audio data, the digital model including a sequence of masking blocks, each masking block including a first component for generating a first mask for extracting speech and a second component for generating a second mask for extracting residual speech masked by the first mask, each of the first mask and the second mask including a mask value estimating the amount of speech present in each frame of the multiple frames and each frequency band of the multiple frequency bands. The method further includes transmitting, to a device, information related to the first mask generated by the sequence of masking blocks. [Effects of the Invention]

[0008] The techniques described herein offer advantages over conventional audio processing techniques. The methods improve audio quality by reducing various types of artifacts and sharpening audio without over-suppressing the audio. The methods utilize deep learning models configured to identify clean audio with low latency and reduce over-suppression of audio with low complexity. Improved audio quality leads to better audio perception and increased user enjoyment of the audio. [Brief explanation of the drawings]

[0009] Example embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numerals refer to like elements:

[0010] [Figure 1] 1 illustrates an example of a networked computer system in which various embodiments may be implemented.

[0011] [Figure 2] FIG. 1 illustrates an example of components of an audio management computer system according to disclosed embodiments.

[0012] [Figure 3] It shows a CGRU block, which consists of a convolutional neural network (CNN) block and a gated recurrent unit (GRU) block.

[0013] [Figure 4] It shows a deep neural network (DNN) consisting of an input CNN block, a CGRU block, and a mask integration block.

[0014] [Figure 5] 1 illustrates an exemplary process performed by an audio management computer system according to some embodiments described herein.

[0015] [Figure 6] 1 is a block diagram illustrating a computer system upon which an embodiment of the present invention may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0016] Description of Example Embodiments In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the example embodiment(s) of the present invention. However, it will be apparent that the example embodiment(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the example embodiment(s).

[0017] The following sections outline embodiments: 1. General overview 2. Example of a computing environment 3. Computer Component Examples 4. Functionality 4.1. Model Training for Speech Enhancement 4.1.1.Feature Extraction Machine Learning Models Perceptual Loss Function 4.2. Model Implementation for Speech Enhancement 5. Process Example 6. Hardware Implementation 7. Extensions and Substitutions **

[0018] 1. General overview

[0019] A system for reducing audio artifacts is disclosed. In some embodiments, the system is programmed to construct a machine learning model including a series of masking blocks. Each masking block receives a feature vector of an audio segment. Each masking block includes a first component that generates a first mask for extracting clean speech and a second component that generates a second mask for extracting residual speech masked by the first mask. Each masking block also generates a specific feature vector based on the first and second masks, which becomes the specific feature vector for the next masking block. The second component, which may include a GRU layer, has lower computational complexity than the first component, which may include multiple CNN layers. The system is further programmed to receive the input feature vector of the input audio segment and execute the machine learning model to obtain an output feature vector of an output audio segment that includes cleaner speech than the input audio segment.

[0020] In one embodiment, the first component includes a first series of CNN blocks. The first series of CNN blocks includes, in an autoencoder structure, CNN blocks with increasing expansion rates followed by decreasing expansion rates and a final CNN block for classification. Each CNN block may include a CNN layer with one or more filters, followed by a batch normalization (BatchNorm) layer and an activation layer. The second component includes a GRU block, which may include a GRU layer followed by a BatchNorm layer and an activation layer. The masking component may also include a third component including a second series of CNN blocks for integrating the first mask and the second mask.

[0021] In some embodiments, the machine learning model includes an input CNN block including a CNN layer with one or more look-ahead filters to enhance the input feature vector. The machine learning model may also include a mask integration block similar to the third component of the masking block. The mask integration block includes a third series of CNN blocks for integrating the first masks generated by the first series of CNN blocks in the series of masking blocks.

[0022] This system provides technical advantages. It addresses the technical problem of improving audio data to enhance speech. It improves audio quality by reducing various types of artifacts and sharpening speech without over-suppressing it. The system utilizes deep learning models that identify clean speech with low latency and reduce over-suppression with low complexity. Improved audio quality leads to better audio perception and a greater user enjoyment of the audio.

[0023] 2. Example of a computing environment

[0024] 1 illustrates an exemplary networked computer system in which various embodiments may be implemented. FIG. 1 is illustrated in simplified schematic form for purposes of clarity, and other embodiments may include more, fewer, or different elements.

[0025] In some embodiments, the networked computer system comprises an audio management server computer 102 (“server”), one or more sensors 104 or input devices, and one or more output devices 110, which are communicatively coupled via direct physical connections or via one or more networks 118.

[0026] In some embodiments, server 102 broadly represents one or more computers, virtual computing instances, and / or instances of applications programmed or configured with data structures and / or database records arranged to host or perform functionality related to audio enhancement. Server 102 may comprise a server farm, a cloud computing platform, a parallel computer, or any other computing facility having sufficient computing power for data processing, data storage, and network communication for the functionality described above.

[0027] In some embodiments, each of the one or more sensors 104 may include a microphone or other digital recording device that converts sound into an electrical signal. Each sensor is configured to transmit detected audio data to the server 102. Each sensor may include a processor or may be integrated into a typical client device, such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device.

[0028] In some embodiments, each of the one or more output devices 110 may include a speaker or other digital playback device that converts electrical signals back into sound. Each output device is programmed to play audio data received from the server 102. Like sensors, the output devices may include a processor or may be integrated into a typical client device, such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device.

[0029] The one or more networks 118 may be implemented by any medium or mechanism that provides for the exchange of data between the various elements of Figure 1. Examples of networks 118 include, but are not limited to, one or more of a cellular network communicatively connected to a data connection to a computing device via a cellular antenna, a near field communication (NFC) network, a local area network (LAN), a wide area network (WAN), the Internet, a terrestrial or satellite link, etc.

[0030] In some embodiments, the server 102 is programmed to receive input audio data corresponding to sounds in a given environment from one or more sensors 104. The input audio data may include multiple frames over time. The server 102 is then programmed to process the input audio data (which typically corresponds to a mixture of speech and noise or other artifacts) to estimate how much speech is present (or detect the amount of speech) in each frame of the input audio data. The server can be programmed to send the final detection results to another device for downstream processing. The server is also programmed to update the input audio data based on the final detection results to generate cleaned-up output audio data that is expected to contain cleaner speech than the input audio data, and send the output audio data to one or more output devices 110.

[0031] 3. Computer Component Examples

[0032] FIG. 2 illustrates example components of an audio management computer system according to disclosed embodiments. This diagram is for illustrative purposes only; the server 102 may include fewer or more functional or storage components. Each of the functional components may be implemented as a software component, a general-purpose or specialized hardware component, a firmware component, or any combination thereof. Each of the functional components may also be coupled to one or more storage components. The storage components may be implemented using any of a relational database, an object database, a flat file system, or a Javascript Object Notation (JSON) store. The storage components may connect to the functional components locally or over a network using program calls, a remote procedure call (RPC) facility, or a messaging bus. Components may or may not be self-contained. Depending on implementation-specific or other considerations, components may be functionally or physically centralized or distributed.

[0033] In some embodiments, server 102 comprises machine learning model training instructions 202, machine learning model execution instructions 206, and communication interface instructions 210. Server 102 also comprises database 220.

[0034] In some embodiments, the machine learning model training instructions 202 enable training of a machine learning model for speech detection and artifact reduction. The machine learning model may include various artificial neural networks (ANNs) or other transformation or classification models. The training may include extracting features from training audio data, training the machine learning model by feeding the given features or extracted features (optionally along with expected model output) to a training framework, and storing the trained machine learning model. The output of the machine learning model may include an estimate of the amount of speech present in each given audio segment or an emphasis of each given audio segment. The training framework may include an objective function designed to mitigate over-suppression of speech.

[0035] In some embodiments, the machine learning model execution instructions 206 enable execution of a machine learning model for speech detection and artifact mitigation. Execution can include extracting features from a new audio segment, feeding the extracted features to a trained machine learning model, and obtaining new output from executing the trained machine learning model. The new output can include an estimate of the amount of speech in the new audio segment or an enhanced version of the new audio segment.

[0036] In some embodiments, the communications interface instructions 210 enable communication with other systems or devices over a computer network. Communication can include receiving audio data or trained machine learning models from audio sources or other systems. Communication can also include transmitting results of speech detection or enhancement to other processing or output devices.

[0037] In some embodiments, database 220 is programmed or configured to manage the storage and access of associated data, such as received audio data, digital models, features extracted from the received audio data, or results of running the digital models.

[0038] 4. Functionality

[0039] 4.1. Model Training for Speech Enhancement

[0040] Data Collection

[0041] Speech signals are typically distorted by various contaminations or artifacts, such as noise and reverberation, caused by the environment and recording equipment. In some embodiments, the server 102 is programmed to build a training dataset of audio segments distorted to various degrees. The audio segments in the training dataset may contain additive artifacts affecting different durations or frequency bands. An example of an approach to blending such artifacts with a clean speech signal can be found in the paper "Multi-task self-supervised learning for robust speech recognition" by Ravanelli et al., which describes an improved version of the problem-agnostic speech encoder (PASE+).

[0042] 4.1.2.Feature Extraction

[0043] A training dataset of audio segments is typically represented in the time domain. In some embodiments, the server 102 is programmed to convert each audio segment, consisting of a waveform spanning multiple frames, into a joint time-frequency (TF) representation using a spectral transform such as a short-term Fourier transform (STFT), a modified discrete Fourier transform (MDFT), or a complex quadratic mirror filter (CQMF). The joint TF representation covers multiple frames and multiple frequency bins.

[0044] In some embodiments, the server 102 is programmed to convert the TF representation into a vector of band-like energies, such as 56 perceptually motivated bands. Each perceptually motivated band is typically located in a frequency range that corresponds to how the human ear processes sound, such as 120 Hz to 2,000 Hz, and capturing data in these perceptually motivated bands means that no loss of audio quality is experienced by the human ear. More specifically, the squared magnitudes of the output frequency bins of the spectral transform are grouped into perceptually motivated bands, where the number of frequency bins per band increases at higher frequencies. The grouping strategy may be "soft," in which some spectral energy leaks between adjacent bands, or "hard," in which there is no leakage between bands. Specifically, when the bin energies of a noisy frame are represented by a column vector of size p × 1, where x is the number of bins, the conversion of band-like energies into a vector is performed by calculating y = W * x. where y is a column vector of size q × 1 representing the band energy of this noisy frame, and W is a banding matrix of size q × p, where q is the number of perceptually motivated bands.

[0045] In some embodiments, the server 102 is then programmed to calculate the logarithm of each band-like energy as a feature value for each frame and each frequency band. Alternatively, the band energy can be used directly as a feature value. In this way, for each joint TF representation, an input feature vector consisting of feature values ​​can be obtained for multiple frames and multiple frequency bands.

[0046] In some embodiments, for supervised learning, the server 102 is programmed to obtain or calculate, for each joint TF representation, an expected mask that indicates the amount of speech present in each frame and each frequency band. The mask may be in the form of the logarithm of the ratio of speech energy to the sum of all energies. The server 102 may include the expected mask in the training dataset.

[0047] Machine Learning Models

[0048] In some embodiments, the server 102 is then programmed to train a machine learning model using the training dataset. Figure 3 illustrates a CGRU block, which consists of a CNN block and a GRU block. All aspects in Figures 3 and 4, including the number of blocks, block types, or parameter values, are shown for illustrative purposes only. The CGRU block 300 receives an input feature vector 316 of an input audio segment and generates an output mask 312 and an output feature vector 320 corresponding to cleaner speech. The input feature vector 316 may be in the form (N, C, F, T), where N indicates a batch size (e.g., the number of audio files), C indicates the number of channels, F indicates a value in the frequency dimension, and T indicates a value in the time dimension. The CGRU block 300 includes a first series of CNN blocks 302 and a GRU block 308, and generates respective masks 304 and 314 for separating additive artifacts from clean speech in the input audio segment while reducing over-suppression of speech. Each of the masks 304 and 314 may also be of the form (N, C, F, T). The CGRU block 300 also includes a second series of CNN blocks 310 that combine the masks 304 and 314 into an output mask 312. As described below, the output mask 312 may be applied to an input feature vector 316 for further speech enhancement. The CGRU block 300 functions as a component of a deep neural network (DNN), as described further below.

[0049] In some embodiments, the first series of CNN blocks 302 is for detecting clean speech. The first series of CNN blocks 302 includes a first subsequence of augmented CNN blocks with increasing augmentation rates (e.g., 1, 3, 9, 27 along the time dimension), followed by a second subsequence of augmented CNN blocks with corresponding decreasing augmentation rates (e.g., 27, 9, 3, 1 along the time dimension), followed by a single tail CNN block. As shown, the first series of CNN blocks 302 receives input feature vectors 316. Each CNN block in the first series receives a feature vector for an audio segment and generates a specific feature vector for an enhanced audio segment. The output of each CNN block in the first series becomes the input for the next CNN block in the first series. The output of each CNN block in the first subsequence is combined with the input of a CNN block with the same augmentation rate in the second subsequence. The combination can be performed by addition or concatenation. Each expanded CNN block uses relatively small filters, such as sixteen 3x3 filters, and the final CNN block uses one 1x1 filter. In other embodiments, the length of the first or second subsequence, the expansion rate, the number of filters, and the size of each filter may vary.

[0050] Thus, the first subseries of CNN blocks with expanding receptive fields encodes feature data characterizing clean speech in the original audio data (finding more and better features), while the second subseries of CNN blocks reconstructs the enhanced audio data. The final CNN block performs a linear projection of the feature map into a summary feature map that indicates how much speech is present in the original audio data. In this way, the first series of CNN blocks projects different levels of discriminative features into a high-resolution space, i.e., the per-band level for each frame, to obtain a dense classification (how much speech is present for each time frame and each frequency band), i.e., the first mask 304.

[0051] In some embodiments, the GRU block 308 is for detecting speech that may have been over-suppressed by the first series of CNN blocks 302. The input feature vector 316 may be integrated with the first mask 304 generated by the first series of CNN blocks (this integration is not shown as a separate block in FIG. 3 ) to generate a residual feature vector 306 that corresponds to the portion of the input feature vector 316 that is not identified as speech by the first mask 304. For example, the inverse transform I(m), e.g., log(1−e m ) can be applied to the corresponding feature values ​​f of the input feature vector, e.g., I(m)+f, to generate the inverse transform of the result of applying the first mask 304 to the input feature vector 316 as the values ​​of the residual feature vector 306.

[0052] In some embodiments, the GRU block 308 then receives the residual feature vector 306 and generates a residual mask 314. The GRU block 308 may be composed of one or more GRU layers followed by a BatchNorm layer followed by an activation layer, as known to those skilled in the art. The BatchNorm layer can accept one-dimensional input (BatchNorm1d). Each GRU layer can contain a fixed number of nodes equal to the number of features in each feature vector. The GRU layer provides a gating mechanism for recurrent neural networks (RNNs), which are often used to process time-series data. Its relatively simple structure makes it particularly suitable for detecting relatively small amounts of residual speech, as it allows for relatively relaxed filtering of non-speech. Its relatively simple structure also helps achieve low complexity in machine learning models. In other embodiments, to further reduce complexity, the GRU layer can be replaced with a simple fully connected layer without recurrent connections. For further simplification, the layer can use one-dimensional filters along the time dimension instead of two-dimensional filters. BatchNorm layers typically serve to fine-tune the output of the previous layer and avoid internal covariate shifts. Activation layers typically serve to limit and maintain the output values ​​and add nonlinearity to the ANN. In certain embodiments, a BatchNorm layer can precede an activation layer.

[0053] In some embodiments, the second sequence of CNN blocks 310 is for generating an output mask 312 that induces cleaner speech than the input audio segment. The second sequence of CNN blocks 310 can include two CNN blocks, where the first CNN block has relatively small filters, such as 16 3x3 filters, and the second CNN block is similar to the last CNN block in the first sequence of CNN blocks. The second sequence of CNN blocks 310 receives the first mask 304 and the residual mask 314 and combines (e.g., concatenates) the two masks to generate the output mask 312. In other embodiments, the length of the second sequence, the number of filters, and the size of each filter may vary.

[0054] Thus, a first CNN block of the second series of CNN blocks 310 identifies a particular pattern from masks 304 and 314, and a second CNN block of the second series of CNN blocks 310 again performs classification to determine an effective way to combine first mask 304 and residual mask 314. Output mask 312 is then applied to input feature vector 316 via feature vector generation 322 to generate output feature vector 320, which is similar to the inverse transformation where first mask 304 is applied to input feature vector 316 to generate residual feature vector 306, as described above.

[0055] In some embodiments, each CNN block in the CGRU block 300 includes a CNN layer, a BatchNorm layer, and an activation layer. The CNN layer consists of a two-dimensional filter for causal convolution. The BatchNorm layer accepts two-dimensional input (BatchNorm2d). The activation layer is similar to that of the GRU block.

[0056] Figure 4 shows a deep neural network including an input CNN block, a CGRU block, and a mask synthesis block. The DNN 400 includes an input CNN block 402, a series of CGRU blocks 404, each corresponding to 300 shown in Figure 3, and a mask synthesis block 406. The DNN 400 receives an original feature vector 412 of an original audio segment and predicts a final mask 418 that is used to extract clean speech from the original audio segment.

[0057] In some embodiments, the input CNN block 402 is for enriching the original feature vector 412. The input CNN block 402 includes a CNN layer, a BatchNorm layer, and an activation layer. In the CNN layer, compared to the CNN layers in each CNN block in the CGRU block, each filter is applied to a smaller number of future frames, such as an additional look-ahead of two frames. Such filters can be referred to as look-ahead filters. Thus, the input CNN block 402 receives the original feature vector 412 and generates a feature map as the refined feature vector 414. The relatively small amount of additional look-ahead helps achieve low latency for the DNN 400 while generating enriched features. In other embodiments, the input CNN block 402 is not present, and the original feature vector 412 is directly received by the CGRU block 404.

[0058] In some embodiments, the first CGRU block in the sequence of CGRU blocks 404 receives the refined feature vector 414, predicts a mask 408 corresponding to 308 in FIG. 3 , and generates a feature vector 410 corresponding to 320 in FIG. 3 . The feature vector 410 then becomes the input to the next CGRU block, and the process continues through the sequence of CGRU blocks 404. As described above, the first sequence of CNN blocks in each CGRU block is intended to remove as much non-speech as possible, but may also remove some speech. The GRU block in each CGRU block is intended to return the removed speech, but may also return non-speech. The repeated configuration of CGRU blocks in the sequence of CGRU blocks 404 is intended to ultimately retain as much clean speech as possible while removing as much non-speech as possible, including various artifacts such as echo, noise, or reverberation. Experiments have shown that a near-equilibrium point can be reached after, for example, four CGRU blocks. In other embodiments, the length of the sequence of CGRU blocks may vary.

[0059] The mask integration block 406 effectively integrates the masks predicted by the sequence of CGRU blocks 404. The mask integration block 406 receives the first masks predicted by the first sequence of CNN blocks in each CGRU block in the sequence of CGRU blocks 404 and generates the final mask 418. The mask integration block 406 includes two CNN blocks, similar to the second sequence of CNN blocks 310 in FIG. 3. The first CNN block identifies specific patterns from the first masks, and the second CNN block again performs classification to determine an effective way to integrate the first masks. While the second sequence of CNN blocks 310 takes two masks to generate the output mask 312, the mask integration block 406 takes four masks (or the number of CGRU blocks) to generate the final mask 418.

[0060] Perceptual Loss Function

[0061] In some embodiments, the server 102 is programmed to train the machine learning model using a suitable optimization method known to those skilled in the art. The optimization method is often iterative in nature and may minimize a loss (or cost) function that measures the error of the current estimate from truth. In the case of an ANN, the optimization method may be stochastic gradient descent, with weights updated using a backpropagation algorithm.

[0062] Traditionally, objective and loss functions such as mean squared error (MSE) do not closely reflect human hearing. Processed speech segments with low MSE do not necessarily have high speech quality or intelligibility. Specifically, objective functions do not distinguish between negative detection errors (false negatives, over-suppression of speech) and positive detection errors (false positives, under-suppression of speech), even though over-suppression of speech can have a greater impact on perception than under-suppression and is often treated differently from under-suppression in speech enhancement applications.

[0063] Over-suppression of speech can be more detrimental to speech quality or intelligibility than under-suppression of speech. Over-suppression of speech occurs when the predicted (estimated) mask value is smaller than the true mask value because less speech is predicted than true, resulting in more speech being suppressed than necessary.

[0064] In some embodiments, an optimization method for training a machine learning model uses a perceptual cost function that works to reduce over-suppression of speech. The perceptual cost function is a nonlinear function that has asymmetric penalties for over-suppression of speech and under-suppression of speech. Specifically, the cost function assigns a greater penalty to negative differences between predicted mask values ​​and true mask values ​​and a lesser penalty to positive differences. Experiments have shown that the perceptual loss function outperforms MSE in reducing over-suppression of speech for, for example, high-frequency fricatives and low-level pauses such as "um" and "uh."

[0065] In some embodiments, the perceptual loss function Loss is defined as follows: diff=y target p -y predicted p (1) Loss=m diff -diff-1 (2) where y target is the target (true) mask value for a certain frame and a certain frequency band, y predicted is the predicted mask value for that frame and that frequency band, m is a tuning parameter that can control the shape of the asymmetry penalty, and p is a power law term or scaling exponent. For example, m can be 2.6, 2.65, 2.7, etc., and p can be 0.5, 0.6, 0.7, etc. predicted or y target is less than 1, so if p is a fractional value that is not too small (e.g., greater than 0.5), then y predicted or y target than larger values ​​of y predictedThe fractional value of p tends to amplify the small values ​​of y target P and y predicted p The difference between target and y predicted The difference between y and y tends to be even larger than that between y and y. predicted A small value of means that the frame contains noise (small values ​​of y target ) and then over-suppression (y predicted This may be the result of a series of steps (leading to even smaller values ​​of y). target and y predicted The difference between target P and y predicted p (Using a too small value for p may result in too frequent amplification), the penalty for over-suppressing such speech becomes greater. Thus, the power law term can be useful for improving over-suppression of speech, especially in the difficult case of noisy frames. The inherent focus on such difficult cases also leads to the possibility of having smaller machine learning models with fewer parameters. The total loss for an audio signal corresponding to multiple frequency bands and multiple frames can be calculated as the sum or average of the loss values ​​across the multiple frequency bands and multiple frames.

[0066] In some embodiments, the perceptual loss function Loss is based on MSE as follows: diff=y target p -y predicted p (3) w=m diff -diff-1 (4) Loss=w*diff 2 (5)

[0067] MSE penalizes positive and negative diff values ​​equally, so negative diff values, which indicate over-suppression of speech, are not penalized more than positive diff values, which indicate under-suppression of speech. In the Loss defined in equation (5), significant under-suppression of speech, which corresponds to a predicted mask value being much lower than the target mask value, is penalized by a large weight, w, and a large diff. 2 (through correspondingly large errors) will be penalized multiple times.

[0068] 4.2. Model Implementation for Speech Enhancement

[0069] In some embodiments, the server 102 is programmed to receive a new audio signal having one or more frames in the time domain. The server 102 then applies the machine learning approach described in Section 4.1 to the new audio signal to generate a prediction mask indicating the amount of speech present in each frame and each frequency band in the corresponding TF representation. This application includes converting the new audio signal into a joint TF representation that initially covers multiple frames and multiple frequency bins.

[0070] In some embodiments, the server 102 is programmed to further generate an improved audio signal for the new audio signal based on the prediction mask. Let the band mask for y (obtained by applying the machine learning approach described in Section 4.1) be given as a column vector m_band of size q×1, where y is a column vector of size q×1 representing the band energy for the original noisy frame and q represents the number of perceptually motivated bands. The conversion to a bin mask is performed by calculating m_bin=W_transpose *m_band, where m_bin is a column vector of size p×1, p represents the number of bins, and W_transpose of size p×q is the transpose of W, a band matrix of size q×p.

[0071] In some embodiments, the server 102 is programmed to achieve noise masking or reduction by applying a bin mask to the original frequency bin magnitudes in the joint TF representation to obtain an estimated clean spectrum. The server 102 can further convert the estimated clean spectrum back into a waveform as an enhanced waveform (relative to the noise waveform) using any method known to those skilled in the art, such as inverse CQMF, which can be communicated via an output device.

[0072] 5. Process Example

[0073] Figure 5 illustrates an example process performed using an audio management computer system according to some embodiments described herein. Figure 5 is shown in a simplified schematic format for purposes of illustrating a clear example; other embodiments may include more, fewer, or different elements connected in various ways. Figure 5 is intended to disclose algorithms, plans, or schematics that can be used to implement one or more computer programs or other software elements that, when executed, cause the functional improvements and technical advances described herein. Furthermore, the flow diagrams herein are described in the same level of detail that those skilled in the art would normally use to communicate with one another using their accumulated skill and knowledge about the algorithms, plans, or specifications that form the basis of the software programs they plan to code or implement.

[0074] In some embodiments, the server 102 is programmed to receive an input waveform in the time domain and convert the input waveform into raw audio data across multiple frequency bins and multiple frames. The server 102 is further programmed to convert the raw audio data into the audio data by grouping the multiple frequency bins into the multiple frequency bands, the joint time-frequency representation having an energy value for each time frame and each frequency band. The joint time-frequency representation having an energy value for each time frame and each frequency band.

[0075] Thus, in step 502, the server 102 is programmed to receive audio data as a joint time-frequency representation across multiple frames and multiple frequency bands.

[0076] In some embodiments, the server 102 is programmed to generate a feature vector from the joint time-frequency representation.

[0077] Then, in step 504, the server 102 is programmed to execute a digital model for detecting speech from the feature vector of the audio data. The digital model includes a series of masking blocks. Each masking block includes a first component for generating a first mask for extracting speech and a second component for generating a second mask for extracting residual speech masked by the first mask. Each of the first and second masks includes a mask value that estimates the amount of speech present in each frame of a plurality of frames and each frequency band of a plurality of frequency bands.

[0078] In some embodiments, the first component includes a sequence of combined CNN blocks with extensions. Each CNN block in the sequence of combined CNN blocks includes a CNN layer, a batch normalization layer, and an activation layer. The first component also includes a CNN layer with a 1×1 filter. In other embodiments, the second component includes a gated recurrent unit (GRU) block including a GRU layer.

[0079] In some embodiments, the first component of a first masking block in the series of masking blocks receives as input the feature vector, and the second component of the first masking block receives an inverse transform of the result of applying the first mask to the feature vector.

[0080] In some embodiments, each masking block comprises a third component including a CNN block configured to combine the first mask and the second mask into an output mask. Each masking block further comprises a fourth component that applies the output mask to a feature vector to generate a particular feature vector. A first masking block in the series of masking blocks receives the feature vector as the particular feature vector. Each subsequent masking block in the series of masking blocks receives as input the particular feature vector generated by the preceding masking block.

[0081] In some embodiments, the digital model further includes an input CNN block including a CNN layer with one look-ahead filter, a batch normalization layer, and an activation layer.

[0082] In step 506, the server 102 is programmed to transmit information related to the first mask generated by the sequence of masking blocks to the device.

[0083] In some embodiments, the digital model further includes a mask integration block including a CNN block that integrates the first mask generated from the series of masking blocks into a final mask. In other embodiments, the server 102 is programmed to generate updated mask values ​​for each frequency bin of the plurality of frequency bins and each frame of the plurality of frames by performing inverse band processing on mask values ​​of the final mask. The server 102 is further programmed to apply the updated mask values ​​to the audio data to generate new output data and convert the new output data into an enhanced waveform.

[0084] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE).

[0085] EEE1. A computer-implemented method for reducing audio artifacts, comprising: receiving, by a processor, audio data as a joint time-frequency representation across a plurality of frames and a plurality of frequency bands; executing, by the processor, a digital model for detecting speech from a feature vector of the audio data; the digital model includes a series of masking blocks; Each masking block comprises a first component for generating a first mask for extracting a speech, and a second component for generating a second mask for extracting a residual speech masked by the first mask; each of the first mask and the second mask includes a mask value that estimates a speech presence in each frame of the plurality of frames and each frequency band of the plurality of frequency bands; transmitting information related to the first mask generated by the sequence of masking blocks to a device; A method comprising:

[0086] EEE2. The computer-implemented method of claim 1, wherein the first component comprises a sequence of coupled convolutional neural network (CNN) blocks with dilation.

[0087] EEE3. The computer-implemented method of claim 2, wherein each CNN block in the sequence of combined CNN blocks includes a CNN layer, a batch normalization layer, and an activation layer.

[0088] EEE4. The computer-implemented method of any of claims 1 to 3, wherein the first component includes a CNN layer having a 1x1 filter.

[0089] EEE5. The computer-implemented method of any of claims 1 to 4, wherein the second component includes a gated recurrent unit (GRU) block including a GRU layer.

[0090] EEE6. The computer-implemented method of any of claims 1 to 5, wherein each masking block comprises a third component including a CNN block configured to combine the first mask and the second mask into an output mask.

[0091] EEE7. Each masking block further comprises a fourth component that applies the output mask to a feature vector to generate a particular feature vector; a first masking block in the series of masking blocks receives the feature vector as the one feature vector; each subsequent masking block in the series of masking blocks receives as input the particular feature vector generated by the preceding masking block; 7. The computer-implemented method of claim 6.

[0092] EEE8. The first component of a first masking block of the sequence of masking blocks receives the feature vector as input; the second component of the first masking block receives the inverse of the result of applying the first mask to the feature vector; A computer-implemented method according to any one of claims 1 to 7.

[0093] EEE9. The computer-implemented method of any of claims 1 to 8, wherein the digital model further comprises an input CNN block including a CNN layer with one look-ahead filter, a batch normalization layer, and an activation layer.

[0094] EEE10. The computer-implemented method of any of claims 1 to 9, wherein the digital model further comprises a mask synthesis block including a CNN block that synthesizes the first mask generated from the sequence of masking blocks into a final mask.

[0095] EEE11. generating an updated mask value for each frequency bin of a plurality of frequency bins and for each frame of the plurality of frames by performing inverse band processing on mask values ​​of the final mask; applying the updated mask value to the audio data to generate new output data; converting the new output data into an enhanced waveform; The computer-implemented method of claim 10, further comprising:

[0096] EEE12. Receiving an input waveform in the time domain; converting the input waveform into raw audio data across a plurality of frequency bins and the plurality of frames; converting the raw audio data into the audio data by grouping the plurality of frequency bins into the plurality of frequency bands; the joint time-frequency representation having an energy value for each time frame and each frequency band; generating the feature vector from the joint time-frequency representation; 12. The computer-implemented method of claim 1, further comprising:

[0097] EEE13. The computer-implemented method of any of claims 1 to 12, further comprising training the digital model using a loss function having a non-linear penalty that penalizes over-suppression of speech more than under-suppression of speech.

[0098] EEE14. A system for reducing excessive suppression of speech, comprising: Memory and one or more processors coupled to the memory and configured to: receiving audio data as a joint time-frequency representation across a plurality of frames and a plurality of frequency bands; implementing a digital model for detecting speech from the feature vectors of the audio data; the digital model includes a series of masking blocks; Each masking block comprises a first component for generating a first mask for extracting a speech, and a second component for generating a second mask for extracting a residual speech masked by the first mask; each of the first mask and the second mask includes a mask value that estimates a speech presence in each frame of the plurality of frames and each frequency band of the plurality of frequency bands; Transmitting information related to the first mask generated by the sequence of masking blocks to a device.

[0099] EEE15. A computer-readable non-transitory storage medium having stored thereon computer-executable instructions which, when executed, implement a method for reducing audio artifacts, said method comprising: receiving, by a processor, audio data as a joint time-frequency representation across a plurality of frames and a plurality of frequency bands; implementing a digital model for detecting speech from the feature vectors of the audio data; The digital model includes a masking block including a first series of CNN blocks that generate a first mask for extracting speech, and a GRU block that generates a second mask for extracting residual speech masked by the first mask; Each CNN block of the first series of CNN blocks includes a CNN layer, and each GRU block includes a GRU layer; each of the first mask and the second mask includes a mask value that estimates a speech presence in each frame of the plurality of frames and each frequency band of the plurality of frequency bands; transmitting information related to the first mask and the second mask; 1. A computer-readable non-transitory storage medium, comprising:

[0100] EEE16. The masking block further comprises an additional block for deriving a specific feature vector from a feature vector using the first mask and the second mask; the digital model includes a sequence of masking blocks including the masking block; a first masking block in the series of masking blocks receives the feature vector; each subsequent masking block in the series of masking blocks receives as input the particular feature vector generated by the preceding masking block; 16. The computer-readable non-transitory storage medium of claim 15.

[0101] EEE17. The computer-readable non-transitory storage medium of claim 16, wherein the digital model includes a mask synthesis block including a CNN block that synthesizes the first masks generated by the sequence of masking blocks into a final mask.

[0102] EEE18. The computer-readable non-transitory storage medium of any of claims 15 to 17, wherein the first series of CNN blocks have increasing expansion rates followed by decreasing expansion rates.

[0103] EEE19. The computer-readable non-transitory storage medium of any of claims 15 to 18, wherein the masking block further includes a CNN block that combines the first mask and the second mask into an output mask.

[0104] EEE20. The computer-readable non-transitory storage medium of any of claims 15 to 19, wherein the digital model further comprises an input CNN block including a CNN layer with one look-ahead filter, a batch normalization layer, and an activation layer.

[0105] 6. Hardware Implementation

[0106] According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented, in whole or in part, using a combination of at least one server computer and / or other computing devices connected using a network, such as a packet data network. The computing device may be hardwired to perform the techniques, or may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) persistently programmed to perform the techniques, or may include at least one general-purpose hardware processor programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to achieve the techniques. The computing device may be a server computer, a workstation, a personal computer, a portable computer system, a handheld device, a mobile computing device, a wearable device, a body-worn or implantable device, a smartphone, a smart appliance, an internetworking device, an autonomous or semi-autonomous device (such as a robot or an unmanned ground or air vehicle), any other electronic device embedded with hardwired and / or program logic to implement the above techniques, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0107] Figure 6 is a block diagram illustrating an example of a computer system in which an embodiment may be implemented. In the example of Figure 6, a computer system 600 and instructions for implementing the disclosed techniques in hardware, software, or a combination of hardware and software are represented diagrammatically, e.g., as boxes and circles, at a level of detail commonly used by those skilled in the art to which this disclosure pertains, to facilitate understanding of computer architecture and computer system implementation.

[0108] Computer system 600 includes an input / output (I / O) subsystem 602, which may include a bus and / or other communication mechanisms for communicating information and / or instructions between components of computer system 600 via electronic signal paths. I / O subsystem 602 may include an I / O controller, a memory controller, and at least one I / O port. The electronic signal paths are represented schematically in the figure, for example, as lines, single-headed arrows, or double-headed arrows.

[0109] At least one hardware processor 604 is coupled to the I / O subsystem 602 for processing information and instructions. The hardware processor 604 may include, for example, a general-purpose microprocessor or microcontroller and / or a special-purpose microprocessor such as an embedded system, a graphics processing unit (GPU), a digital signal processor, or an ARM processor. The processor 604 may include an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

[0110] Computer system 600 includes memory 606, consisting of one or more units (e.g., main memory), coupled to I / O subsystem 602 for electronically and digitally storing data and instructions to be executed by processor 604. Memory 606 may include volatile memory, such as various forms of random access memory (RAM) or other dynamic storage devices. Memory 606 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 604. Such instructions, when stored on a non-transitory computer-readable storage medium accessible to processor 604, may turn computer system 600 into a special-purpose machine customized to perform the operations specified in the instructions.

[0111] Computer system 600 further includes non-volatile memory (such as read-only memory (ROM) 608 or other static storage device) coupled to I / O subsystem 602 for storing information and instructions for processor 604. ROM 608 may include various forms of programmable ROM (PROM), such as erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM). A unit of persistent storage 610, which may include various forms of non-volatile random access memory (NVRAM), such as flash memory, or solid-state storage, a magnetic disk, or an optical disk, such as a CD-ROM or DVD-ROM, may be coupled to I / O subsystem 602 for storing information and instructions. Storage 610 is one example of a non-transitory computer-readable medium that may be used to store instructions and data that, when executed by processor 604, cause computer-implemented methods to perform the techniques described herein.

[0112] The instructions in memory 606, ROM 608, or storage 610 may include one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps. The instructions may include operating systems and / or system software; one or more libraries supporting multimedia, programming, or other functionality; data protocol instructions or stacks implementing TCP / IP, HTTP, or other communications protocols; file processing instructions for interpreting and rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a graphical user interface (GUI), command line interface, or text user interface; application software such as an office suite, Internet access application, design and manufacturing application, graphics application, audio application, software engineering application, educational application, game, or other application. The instructions may implement a web server, web application server, or web client. The instructions may be organized as a presentation layer, an application layer, and a data storage layer, such as a relational database system using structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0113] Computer system 600 may be connected to at least one output device 612 via I / O subsystem 602. In one embodiment, output device 612 is a digital computer display. Examples of displays that may be used in various embodiments include a touchscreen display or a light-emitting diode (LED) display or a liquid crystal display (LCD) or an electronic paper display. Computer system 600 may include other types of output device 612 instead of, or in addition to, a display device. Examples of other output device 612 include a printer, a ticket printer, a plotter, a projector, a sound or video card, a speaker, a buzzer or piezoelectric device or other audible device, a lamp or LED or LCD indicator, a tactile device, an actuator, or a servo.

[0114] At least one input device 614 is connected to the I / O subsystem 602 for communicating signals, data, command selections, or gestures to the processor 604. Examples of input device 614 include touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and / or various types of sensors such as force sensors, motion sensors, thermal sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors, and / or various types of transceivers such as wireless, radio frequency (RF) or infrared (IR) transceivers such as cellular or Wi-Fi, and global positioning system (GPS) transceivers.

[0115] , Another type of input device is the control device 616. The control device 616 may perform cursor control or other automated control functions, such as navigating a graphical interface on a display screen, instead of or in addition to input functions. The control device 616 may be a touchpad, mouse, trackball, or cursor direction keys for communicating directional information and command selections to the processor 604 and for controlling cursor movement on the display 612. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to locate a position in a plane. Another type of input device is a wired, wireless, or optical control device, such as a joystick, wand, console, steering wheel, pedals, gear shift mechanism, or other type of control device. The input device 614 may include a combination of multiple different input devices, such as a video camera and a depth sensor.

[0116] In another embodiment, computer system 600 may comprise an Internet of Things (IoT) device that omits one or more of output device(s) 612, input device(s) 614, and control device(s) 616. Alternatively, in such an embodiment, input device(s) 614 may comprise one or more cameras, motion detectors, thermometers, microphones, earthquake detectors, other sensors or detectors, measurement devices, or encoders, and output device(s) 612 may comprise a dedicated display, such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator, or a servo.

[0117] If computer system 600 is a mobile computing device, input device 614 may include a Global Positioning System (GPS) receiver connected to a GPS module that can triangulate against multiple GPS satellites to determine and generate geolocation or position data, such as latitude-longitude values, for the geophysical location of computer system 600. Output device 612 may include hardware, software, firmware, and interfaces for generating position report packets, notifications, pulse or heartbeat signals, or other recurring data transmissions that identify the location of computer system 600, alone or in combination with other application-specific data, to host 624 or server 630.

[0118] Computer system 600 may implement the techniques described herein using customized hardwired logic, at least one ASIC or FPGA, firmware, and / or program instructions or logic that, when loaded and used or executed in conjunction with the computer system, cause or program the computer system to operate as a special-purpose machine. According to one embodiment, the techniques are performed by computer system 600 in response to processor 604 executing at least one sequence of at least one instruction contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0119] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage 610. Volatile media include, for example, dynamic memory, such as memory 606. Common forms of storage media include, for example, hard disks, solid-state drives, flash drives, magnetic data storage media, any optical or physical data storage media, memory chips, etc.

[0120] Storage media are distinct from, but may be used in conjunction with, transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and fiber optics, including wires such as a bus in I / O subsystem 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0121] Various forms of media may be involved in carrying at least one sequence of at least one instruction to processor 604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its own dynamic memory and transmit the instructions over a communications link, such as a modem, fiber optic or coaxial cable, or a telephone line. Data on the communications link may be received by a modem or router local to computer system 600 and converted so as to be readable by computer system 600. For example, data carried by a radio or optical signal may be received by a receiver, such as a radio frequency antenna or infrared detector, and provided by appropriate circuitry to I / O subsystem 602 (e.g., placing the data on a bus). I / O subsystem 602 carries the data to memory 606. The data is retrieved from memory 606 by processor 604, and the instructions are executed. The instructions received by memory 606 may optionally be stored on storage 610 either before or after execution by processor 604.

[0122] Computer system 600 also includes a communications interface 618 coupled to I / O subsystem 602. Communications interface 618 provides a two-way data communication coupling to a network link 620, which is directly or indirectly connected to at least one communications network, such as a network 622 or a public or private cloud on the Internet. For example, communications interface 618 may be an Ethernet networking interface, an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing a data communication connection to a corresponding type of communications line, such as an Ethernet cable or any type of metallic or fiber optic line or telephone line. Network 622 broadly represents a LAN, a WAN, a campus network, an internetwork, or any combination thereof. Communications interface 618 may comprise a LAN card providing a data communication connection to a compatible LAN, or a wired cellular radiotelephone interface for transmitting or receiving cellular data in accordance with a cellular radiotelephone wireless networking standard, or a wired satellite radio interface for transmitting or receiving digital data in accordance with a satellite wireless networking standard. In any such implementation, communication interface 618 sends and receives electrical, electromagnetic or optical signals over signal paths that carry digital data streams representing various types of information.

[0123] Network link 620 typically provides electrical, electromagnetic, or optical data communication to other data devices directly or through at least one network, for example using satellite, cellular, Wi-Fi, or Bluetooth technology. For example, network link 620 may provide a connection through network 622 to a host computer 624.

[0124] Further, network link 620 may provide connectivity through network 622 or to other computing devices through internetworking devices and / or computers operated by an Internet Service Provider (ISP) 626. ISP 626 provides data communication services through a worldwide packet data communication network represented as Internet 628. Connected to Internet 628 may be a server computer 630. Server 630 broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or a computer running a containerized program system such as DOCKER or KUBERNETES. Server 630 may represent an electronic digital service implemented using two or more computers or instances and accessed and used by sending a web service request, a uniform resource locator (URL) string with parameters in an HTTP payload, an application programming interface (API) call, an app service call, or other service call. Computer system 600 and server 630 may form elements of a distributed computing system that includes other computers, processing clusters, server farms, or other configurations of computers that cooperate to perform tasks or run applications or services. The server 630 may comprise one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls, which may be organized as one or more computer programs, operating system services, or application programs, including mobile apps.The instructions may include operating system and / or system software; one or more libraries supporting multimedia, programming, or other functionality; data protocol instructions or stacks implementing TCP / IP, HTTP, or other communications protocols; file formatting instructions for interpreting or rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a GUI, command line interface, or text user interface; and application software such as an office suite, Internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or other applications. Server 630 may comprise a web application server hosting a presentation layer, an application layer, and a data storage layer such as a relational database system using Structured Query Language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0125] Computer system 600 can send messages and receive instructions, including data and program code, through the network(s), network link 620 and communication interface 618. In the Internet example, a server 630 might transmit a requested code for an application program through Internet 628, ISP 626, local network 622 and communication interface 618. The received code may be executed by processor 604 as it is received, and / or stored in storage 610, or other non-volatile storage for later execution.

[0126] Execution of the instructions described in this section may implement a process in the form of a running computer program instance, which consists of program code and its current operations. Depending on the operating system (OS), a process may consist of multiple threads of execution that execute instructions simultaneously. In this sense, a computer program is a passive collection of instructions, while a process may be the actual execution of those instructions. Multiple processes may relate to the same program. For example, opening multiple instances of the same program often means that two or more processes are running. Multitasking may be implemented to allow multiple processes to share the processor 604. Although each processor 604 or each core of that processor executes one task at a time, the computer system 600 may be programmed to implement multitasking to allow each processor to switch between multiple running tasks without having to wait for each task to finish. In one embodiment, switching may occur when a task performs an input / output operation, when the task indicates that it is available to switch, or upon a hardware interrupt. Time sharing may be implemented to enable fast response for interactive user applications by performing rapid context switching to allow multiple processes to appear to be running simultaneously. In one embodiment, for safety and reliability, the operating system may prevent direct communication between independent processes and provide a strictly mediated and controlled inter-process communication facility.

[0127] 7. Extensions and Substitutions

[0128] In the above, the disclosed embodiments have been described with reference to many specific details that may vary from implementation to implementation. Thus, the specification and drawings should be considered in an illustrative, not a limiting sense. What is the sole and exclusive indication of the scope of the present disclosure, and what the applicants intend to be the scope of the present disclosure, is the literal and equivalent scope of the set of claims issuing from this application, including any subsequent amendments, in the specific form in which such claims arise.

Claims

1. 1. A computer-implemented method for reducing audio artifacts, comprising: receiving, by a processor, audio data as a joint time-frequency representation across a plurality of frames and a plurality of frequency bands; executing, by the processor, a digital model for detecting speech from a feature vector of the audio data; the digital model includes a series of masking blocks; Each masking block comprises a first component for generating a first mask for extracting a speech, and a second component for generating a second mask for extracting a residual speech masked by the first mask; each of the first mask and the second mask includes a mask value that estimates a speech presence in each frame of the plurality of frames and each frequency band of the plurality of frequency bands; transmitting information related to the first mask generated by the sequence of masking blocks to a device; A method comprising:

2. The computer-implemented method of claim 1 , wherein the first component comprises a sequence of coupled convolutional neural network (CNN) blocks with dilation.

3. 3. The computer-implemented method of claim 2, wherein each CNN block in the sequence of combined CNN blocks includes a CNN layer, a batch normalization layer, and an activation layer.

4. The computer-implemented method of claim 1 , wherein the first component includes a CNN layer having a 1×1 filter.

5. The computer-implemented method of claim 1 , wherein the second component includes a gated recurrent unit (GRU) block that includes a GRU layer.

6. 6. The computer-implemented method of claim 1, wherein each masking block comprises a third component including a CNN block configured to combine the first mask and the second mask into an output mask.

7. each masking block further comprising a fourth component that applies the output mask to a feature vector to generate a particular feature vector; a first masking block of the series of masking blocks receives the feature vector as the one feature vector; each subsequent masking block in the series of masking blocks receives as input the particular feature vector generated by the preceding masking block; 7. The computer-implemented method of claim 6.

8. the first component of a first masking block of the sequence of masking blocks receives the feature vector as an input; the second component of the first masking block receives the inverse of the result of applying the first mask to the feature vector; A computer-implemented method according to any one of claims 1 to 7.

9. 9. The computer-implemented method of claim 1, wherein the digital model further comprises an input CNN block including a CNN layer with one look-ahead filter, a batch normalization layer, and an activation layer.

10. 10. The computer-implemented method of claim 1, wherein the digital model further comprises a mask synthesis block including a CNN block that synthesizes the first mask generated from the sequence of masking blocks into a final mask.

11. performing inverse band processing on mask values ​​of the final mask to generate updated mask values ​​for each frequency bin of a plurality of frequency bins and each frame of the plurality of frames; applying the updated mask value to the audio data to generate new output data; converting the new output data into an enhanced waveform; The computer-implemented method of claim 10 further comprising:

12. receiving an input waveform in the time domain; converting the input waveform into raw audio data across a plurality of frequency bins and the plurality of frames; converting the raw audio data into the audio data by grouping the plurality of frequency bins into the plurality of frequency bands; the joint time-frequency representation having an energy value for each time frame and each frequency band; generating the feature vector from the joint time-frequency representation; The computer-implemented method of claim 1 , further comprising:

13. 13. The computer-implemented method of claim 1, further comprising training the digital model using a loss function having a non-linear penalty that penalizes over-suppression of speech more than under-suppression of speech.

14. 1. A system for reducing over-suppression of speech, comprising: Memory and one or more processors coupled to the memory; Equipped with The one or more processors: receiving audio data as a joint time-frequency representation across a plurality of frames and a plurality of frequency bands; implementing a digital model for detecting speech from the feature vectors of the audio data; the digital model includes a series of masking blocks; Each masking block comprises a first component for generating a first mask for extracting a speech, and a second component for generating a second mask for extracting a residual speech masked by the first mask; each of the first mask and the second mask includes a mask value that estimates a speech presence in each frame of the plurality of frames and each frequency band of the plurality of frequency bands; transmitting information related to the first mask generated by the sequence of masking blocks to a device; A system configured to run

15. 1. A computer-readable non-transitory storage medium having computer-executable instructions stored thereon that, when executed, implements a method for reducing audio artifacts, the method comprising: receiving, by a processor, audio data as a joint time-frequency representation across a plurality of frames and a plurality of frequency bands; implementing a digital model for detecting speech from the feature vectors of the audio data; The digital model includes a masking block including a first series of CNN blocks that generate a first mask for extracting speech, and a GRU block that generates a second mask for extracting residual speech masked by the first mask; Each CNN block of the first series of CNN blocks includes a CNN layer, and the GRU block includes a GRU layer; each of the first mask and the second mask includes a mask value that estimates a speech presence in each frame of the plurality of frames and each frequency band of the plurality of frequency bands; transmitting information related to the first mask and the second mask; 1. A computer-readable non-transitory storage medium, comprising:

16. the masking block further comprises an additional block for deriving a specific feature vector from a feature vector using the first mask and the second mask; the digital model includes a sequence of masking blocks including the masking block; a first masking block in the series of masking blocks receives the feature vector; 16. The computer-readable non-transitory storage medium of claim 15, wherein each subsequent masking block in the series of masking blocks receives as input the particular feature vector produced by a preceding masking block.

17. 17. The computer-readable non-transitory storage medium of claim 16, wherein the digital model includes a mask synthesis block including a CNN block that synthesizes the first masks generated by the sequence of masking blocks into a final mask.

18. 18. The computer-readable non-transitory storage medium of claim 15, wherein the first series of CNN blocks have increasing expansion rates followed by decreasing expansion rates.

19. 19. The computer-readable non-transitory storage medium of claim 15, wherein the masking block further includes a CNN block that combines the first mask and the second mask into an output mask.

20. 20. The computer-readable non-transitory storage medium of claim 15, wherein the digital model further comprises an input CNN block including a CNN layer with one look-ahead filter, a batch normalization layer, and an activation layer.

Images