Enhancing coded speech based on deep generative models

Abstract

A system for generating enhanced speech data using robust speech features is disclosed. In some embodiments, the system is programmed to generate a feature vector set from given coded audio data including contaminated speech using a self-supervised deep learning model. The system is further programmed to generate enhanced speech data corresponding to clean speech from the feature vector set using a generative deep learning model.

Description

[Technical field]

[0001] [Related Applications] This application claims priority to Spanish Patent Application No. P202230148, filed February 23, 2022, and U.S. Provisional Application No. 63 / 431,590, filed December 9, 2022, all of which are incorporated herein by reference.

[0002] [Technical field] This application relates to speech processing and machine learning. [Background technology]

[0003] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued, and thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art by virtue of their inclusion in this section.

[0004] Speech coding first found its main application in secure communications, and later enabled low-cost mobile and Internet communications. As the bit rates of speech codecs are continually decreasing, the decoded speech suffers from a variety of quality problems. Addressing such quality problems would be useful. Summary of the Invention

[0005] A computer-implemented method for recovering clean speech from coded audio data, the method including obtaining coded audio data comprising a first set of frames, the method further including extracting a set of feature vectors from the coded audio data using a self-supervised deep learning model including a neural network, the set of feature vectors each extracted from the first set of frames, the method further including generating enhanced speech data comprising a second set of frames from the set of feature vectors using a generative deep learning model including a neural network, the enhanced speech data corresponding to the clean speech in the coded audio data.

[0006] The techniques described herein are advantageous over conventional audio processing techniques. For example, the method allows for generating improved speech data from real coded audio data, which often contains polluted speech. The generation is based on robust features obtained from self-supervised learning of speech characteristics at various levels, from low-level raw audio characteristics to high-level speaker-related characteristics, which helps to achieve high quality in the improved speech data despite distortion and coding. The improved speech data is efficiently improved using a relatively compact deep learning model that can be executed in real time as the raw speech signal is generated. The improved speech data results in a better perception of the audio and a better user enjoyment of the audio. [Brief description of the drawings]

[0007] Exemplary embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the accompanying drawings in which like reference symbols represent similar elements and in which:

[0008] [Figure 1] 1 illustrates an exemplary networked computer system in which various embodiments may be implemented.

[0009] [Diagram 2]1 illustrates exemplary components of a voice management computer system according to disclosed embodiments.

[0010] [Diagram 3] 1 illustrates the process of training a feature extraction model and training a voice enhancement model.

[0011] [Figure 4] 1 illustrates an exemplary process performed by a voice management computer system according to some embodiments disclosed herein.

[0012] [Diagram 5] FIG. 1 is a block diagram illustrating a computer system upon which one embodiment of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiments of the present invention. It will be apparent, however, that the exemplary embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the exemplary embodiments.

[0014] The embodiments are described in the following sections according to the following overview. 1. Overview 2. Exemplary Computing Environment 3. Exemplary Computer Components 4. Functional Description 4.1. Training the feature extraction model 4.1.1. Data Distortion 4.1.2. Data Downsampling 4.1.3. Feature Extraction Model 4.2. Training the Audio Enhancement Model Data Collection 4.2.2. Audio Enhancement Model 4.3. Running the Deep Learning Model 5. Exemplary Processing 6. Hardware Implementation 7. Extensions and Substitutions

[0015] 1. Overview

[0016] A system for generating enhanced speech data using robust speech features is disclosed. In some embodiments, the system is programmed to generate a feature vector set from given coded audio data including contaminated speech using a self-supervised deep learning model. The system is further programmed to generate enhanced speech data corresponding to clean speech from the feature vector set using a generative deep learning model.

[0017] In some embodiments, the system is programmed to train a self-supervised deep learning model. The training includes using certain audio features of the clean speech signal to calculate weights of an encoder, which is configured to generate a feature vector set from the contaminated coded speech signal. The audio features can include low-level audio features, such as energy or pitch, or high-level audio features, such as speaker accent. The speech signal represents a digitized audio waveform that includes speech. The digitization results in one or more frames, each frame including one or more samples corresponding to a particular sampling rate. The distortion can be caused by extraneous or non-speech audio in the environment, such as noise or reverberation, or recording equipment limitations, such as clipping. The coding is brought about by the application of a compression algorithm with a target bit rate. The trained self-supervised deep learning model is expected to generate, for each frame of the contaminated coded speech signal, a robust feature vector set that characterizes the corresponding frame of the clean speech signal.

[0018] In some embodiments, the clean audio signal is at a particular sampling rate, but the system is programmed to downsample the coded audio signal derivable from the clean audio signal to form a training data set for the self-supervised deep learning model. For example, an audio signal at a sampling rate of 48 kHz can be downsampled to an audio signal at a sampling rate of 16 kHz. The downsampled audio signal is expected to still capture key audio features that can be used to restore the 48 kHz clean signal.

[0019] In some embodiments, the system is programmed to train a generative deep learning model. The training includes calculating weights of a recurrent neural network using a clean speech signal, and the recurrent neural network is configured to generate an improved speech signal from a feature vector set generated by the self-supervised deep learning model from the contaminated coded speech signal. The trained generative deep learning model is expected to generate an improved speech signal of a given sampling rate, such as 48 kHz, even if the feature vector set is generated from a speech signal of a lower sampling rate, such as 16 kHz.

[0020] In some embodiments, given a new speech signal coded using any coding algorithm for any bit rate, the system is configured to run the trained self-supervised deep learning model on the new speech signal to generate a set of feature vectors, and the system is configured to further run the trained generative deep learning model on the set of feature vectors to generate an improved speech signal that corresponds to the decoded clean speech signal.

[0021] The system provides technical advantages. As the bit rate of speech codecs is continuously reduced, the decoded speech suffers from bandwidth loss, spectral holes, or other quality issues. The system addresses the technical problem that the quality of the decoded speech signal is unpredictable due to the bit rate reduction of the coding process. The system allows for the generation of improved speech data from actual coded audio data, which often contains contaminated speech. The generation is based on robust features obtained from self-supervised learning of speech characteristics at various levels, from low-level raw audio characteristics to high-level speaker-related characteristics, which helps to achieve high quality in the improved speech data despite distortion and coding. The improved speech data is efficiently improved using a relatively compact deep learning model that can be executed in real time as the raw speech signal is generated. The improved speech data results in a better perception of the audio and a better user enjoyment of the audio.

[0022] 2. Exemplary Computing Environment

[0023] Figure 1 illustrates an exemplary networked computer system in which various embodiments may be implemented. Figure 1 is illustrated in a simplified schematic form for clarity of illustration, and other embodiments may include more, fewer, or different elements.

[0024] In some embodiments, the networked computer system includes an audio management server computer 102 ("server"), one or more sensors 104 or input devices, and one or more output devices 110, which are communicatively coupled via direct physical connections or via one or more networks 118.

[0025] In some embodiments, server 102 broadly represents one or more computers, virtual computing instances, and / or instances of applications programmed or configured with data structures and / or database records configured to host or perform functions related to restoring a clean audio signal from a corrupted coded audio signal. Server 102 may include a server farm, a cloud computing platform, a parallel computer, or any other computing facility having sufficient computational power in data processing, data storage, and network communications for the functions described above.

[0026] In some embodiments, each of the one or more sensors 104 can include a microphone or another digital recording device that converts sound into an electrical signal. Each sensor is configured to transmit raw or processed audio data to the server 102. Each sensor can include a processor or be integrated into a typical client device, such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device, where a processor coupled to the sensor can perform initial processing of the audio data.

[0027] In some embodiments, each of the one or more output devices 110 may include a speaker or another digital playback device that converts the electrical signals back into audio. Each output device is programmed to play audio data received from the server 102. As with the sensors, the output devices may include a processor or may be integrated into a typical client device, such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device, where a processor coupled to the sensor may perform subsequent processing of the audio data.

[0028] The one or more networks 118 may be implemented by any medium or mechanism that provides for the exchange of data between the various elements of Figure 1. Examples of networks 118 include, but are not limited to, one or more of a cellular network communicatively coupled with a data connection to a computing device via a cellular antenna, a Near Field Communication (NFC) network, a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, terrestrial or satellite links, and the like.

[0029] In some embodiments, the server 102 is programmed to receive input audio data from one or more sensors 104 corresponding to sounds in a given environment. The input audio data may include multiple frames over time. The server 102 is then programmed to process the input audio data, which is then coded and typically has a mixture of speech and noise, to generate improved audio data corresponding to clean speech. The server may also be programmed to send output audio data to one or more output devices.

[0030] 3. Exemplary Computer Components

[0031] FIG. 2 illustrates exemplary components of a voice management computer system according to disclosed embodiments. The diagram is for illustrative purposes only, and server 102 may include fewer or more functional or storage components. Each functional component may be implemented as a software component, a general-purpose or special-purpose hardware component, a firmware component, or any combination thereof. Each functional component may also be coupled with one or more storage components (not shown). The storage components may be implemented using either a relational database, an object database, a flat file system, or a Javascript Object Notation (JSON) store. The storage components may connect to the functional components locally or over a network using program calls, remote procedure call (RPC) functions, or a messaging bus. Components may or may not be self-contained. Depending on implementation-specific or other considerations, components may be functionally or physically centralized or distributed.

[0032] In some embodiments, the server 102 includes feature extraction model training instructions 202, speech enhancement model training instructions 206, model execution instructions 208, and communication interface instructions 210. The server 102 also includes a database 220.

[0033] In some embodiments, the feature extraction model training instructions 202 enable training of a deep learning model to extract audio features corresponding to a clean speech signal from the coded contaminated speech signal. The deep learning model includes various neural networks or other self-supervised models capable of learning features from the clean speech signal.

[0034] In some embodiments, the speech enhancement model training instructions 206 enable training of a deep learning model to generate an improved speech signal from the set of audio features generated by the feature extraction model. The deep learning model includes various neural networks or other generative models capable of generating the improved speech signal from audio features corresponding to a clean speech signal.

[0035] In some embodiments, the model execution instructions 208 enable execution of a deep learning model to generate improved speech data from the contaminated coded speech data. The execution may include executing a feature extraction model on the coded speech data including a mixture of speech and non-speech to generate a feature vector set. The execution may further include executing a speech enhancement model on the feature vector set to generate improved speech data corresponding to clean speech.

[0036] In some embodiments, the communication interface instructions 210 enable communication with other systems or devices over a computer network. The communication can include receiving audio data or a trained deep learning model from an audio source or other system. The communication can also include transmitting enhanced audio data to other processing devices or output devices.

[0037] In some embodiments, database 220 is programmed or configured to manage the storage and access of associated data, such as audio data including clean, corrupted, coded, or downsampled audio signals, digital models, features extracted from the audio data, or results of running digital models.

[0038] 4. Functional Description

[0039] Speech signals, i.e., digital audio data that includes speech and is typically represented as a waveform in the time domain, contain a rich set of acoustic and linguistic features, ranging from individual lexical units such as phonemes or words to speaker characteristics such as the speaker's intention or emotional state. However, such features of speech are not adequately captured by low-level signal processing features such as the amplitude of the wave signal, logarithmic Mel spectrogram, or Mel frequency cepstral coefficients (MFCCs). Coded speech has already lost information. Recovering the original speech using only low-level signal processing features extracted from coded speech can be particularly challenging, as they do not fully characterize the speech signal.

[0040] Thus, in some embodiments, the server uses a feature extraction model, which is a self-supervised deep learning model, to identify characteristics or extract features from the coded speech, and then uses the extracted features to build a speech enhancement model, which is a generative deep learning model.

[0041] 4.1. Training the feature extraction model

[0042] 4.1.1. Data Distortion

[0043] Audio signals are typically distorted by various contaminations caused by the environment or the recording equipment, such as reverberation, additive noise, overlapping voices, time or frequency masking, clipping, etc. Audio coding introduces yet another level of distortion.

[0044] FIG. 3 illustrates the process of training the feature extraction model and training the speech enhancement model. In some embodiments, the server 102 expects a feature extraction model that is robust to speech distortion. Thus, the server 102 prepares a first data set of speech signals, such as 306, distorted to various degrees for the feature extraction model. The speech signals in the first data set may include contaminations that affect different durations or frequency bands. An example of an approach to introduce such contaminations to a clean speech signal, such as 304, can be found in the paper on an improved version of the problem-agnostic speech encoder (PASE+) by Ravanelli et al., entitled "Multi-task self-supervised learning for Robust Speech Recognition." The speech signals in the first data set may also be subject to different codecs, such as Linear Predictive Coding (LPC) or Modified Discrete Cosine Transform (MDCT), with different bit rates, such as bit rates between 32 kbps and 256 kbps. For training purposes, the speech signals may be processed using an Advanced Audio Coding (AAC) codec with different bit rates. Bit rates range from 16kbps to 64kbps.

[0045] 4.1.2. Data Downsampling

[0046] A sampling rate of 48 kHz for digital speech is common in today's multimedia communications. Lower sampling rates can be used to train feature extraction models, based on the reasonable assumption that acoustic and linguistic content such as phonemes and pitch reside primarily in the low-frequency domain. Using a lower sampling rate can reduce the amount of data required and can also improve the discoverability of the required data. Further assumptions include that the low-frequency domain is generally below 8 kHz.

[0047] Thus, in some embodiments, the server 102 selects a lower sampling rate between 8 kHz and 48 kHz, for example 16 kHz. The server 102 then includes the 16 kHz audio signal in a first data set and uses the first data set as a first training set. Alternatively, the server 102 downsamples the 48 kHz audio signal in the first data set to obtain a first training set with a lower sampling rate that is much larger than the first data set. Although the feature extraction model is mainly built from the training data with the lower sampling rate, the feature extraction model operates on and is therefore used to extract features from the original real data with a higher sampling rate, as will be further described below. Experiments show that the feature extraction model built from the first training set with a sampling rate of 16 kHz can restore the clean 48 kHz audio signal with high quality.

[0048] 4.1.3. Feature Extraction Model

[0049] In some embodiments, the server 102 constructs a feature extraction model, such as 320, from the first training set. The server 102 can construct a PASE+ model as described in Ravanelli et al. PASE+ uses a self-supervised learning approach to extract speech information at various levels, such as phonemes or speaker emotion. PASE+ is a deep neural network that takes as input a distorted speech signal, such as a waveform, and produces as output a set of high-dimensional feature vectors that are expected to characterize the corresponding clean speech signal, with each feature for one frame of the distorted speech signal. For example, each frame may be 10 ms long with 480 samples for a sampling rate of 48 kHz. Thus, the server 102 uses each distorted speech signal in the first training set to build the PASE+ model, with the goal of generating a set of feature vectors, such as 308, that characterize the corresponding clean speech without distortion.

[0050] Specifically, PASE+ includes an encoder (including a quasi-recurrent neural network for learning long-term dependencies between time steps) that extracts a group of features as a feature vector from frames of a distorted speech signal. PASE+ also includes a number of workers. Each worker is a small feedforward neural network that accomplishes a self-supervised signal transformation task corresponding to known speech characteristics. For example, if the signal transformation task corresponds to MFCCs, a worker constructs “target” MFCC values ​​from a clean speech signal that corresponds to the distorted speech signal (hence the flow from 304 to 320, for example) and determines the loss of the current feature vector from the target MFCC values. The signal transformation task is considered self-supervised rather than supervised in that no classification or label of the ground truth (e.g., the clean speech signal) is given separately, but the characteristics of the ground truth can and will be calculated automatically from the ground truth. The total loss of the current feature vector is calculated based on the losses determined by all the workers.

[0051] Experimental results show that the distortions introduced to the clean speech signals tend to increase the training data with more diversity and further force the training data to be denoised in order to learn distortion-invariant and robust features. Using the first training set of distorted signals, workers are trained on the encoder to determine the encoder weights. As will be further discussed in Section 4.3, given a real speech signal, it is only necessary to run the trained encoder to obtain the corresponding group of robust features.

[0052] In some embodiments, the worker's tasks can relate to sampling strategies used to capture signal processing features such as filter banks, prosody (interpolated logarithm of fundamental frequency, voiced / unvoiced probability, zero-crossing rate, and energy), log Mel spectrograms, or MFCCs, or speaker characteristics such as local or global information maxima, as also discussed in Ravanelli et al. In this way, PASE+ provides a deep and compact representation of different levels of speech abstraction, from low-level spectral information to high-level speaker information.

[0053] In some embodiments, the server 102 constructs an alternative feature extraction model instead of the PASE+ model. To utilize the first training set, the alternative feature extraction model may also employ a self-supervised or unsupervised learning approach to extract speech information at various levels. One example of an alternative feature extraction model is contrastive predictive coding (CPC), which uses a contrastive loss function that relies on encoding positive samples.

[0054] 4.2. Training the Audio Enhancement Model

[0055] Data Collection

[0056] In some embodiments, the server 102 starts with a sampling rate that is common in today's multimedia communications. The server 102 prepares a second data set of speech signals, such as 334, corrupted to various degrees for the speech enhancement model. The speech signals in the second data set may also be according to different codecs with different bit rates. The server 102 then downsamples the speech signals in the second data set to obtain a downsampled data set with a lower sampling rate, such as 16 kHz. The server then runs the downsampled data set through the trained feature extraction model to obtain a second training set of feature vector sets, such as 338, for the speech enhancement model.

[0057] 4.2.2. Audio Enhancement Model

[0058] In some embodiments, the server 102 constructs an audio enhancement model, such as 350, from the second training set. The server 102 can construct a WaveRNN model as described in the paper entitled "Efficient Neural Audio Synthesis" by Kalchbrenner et al. WaveRNN uses a sequential process to efficiently generate high quality samples. WaveRNN is a deep neural network that takes as input a set of feature vectors, one for each frame of the original audio signal, that are expected to characterize a corresponding clean audio signal, and generates as output a corresponding improved audio signal that is expected to correspond to the clean audio signal. Thus, the server 102 can use the set of feature vectors generated from the distorted audio signal by the feature extraction model to construct a WaveRNN model for each distorted audio signal, with the goal of obtaining an improved audio signal, such as 340, that is free of distortions present in the distorted audio signal.

[0059] Specifically, WaveRNN includes a conditional network and a recurrent network. The conditional network consists of a pair of convolutional networks, a residual network and an upsampling network, operating in parallel. The residual network, which undergoes increasing dilation through blocks of learned transformations, is able to map a feature vector set into a latent representation, which is then split into multiple parts that are used as inputs to the subsequent recurrent network. At the same time, the feature vector set passes through an upsampling network to generate a second set of feature vectors that match the temporal size of the original audio signal. The outputs of these two convolutional networks are concatenated to form an output feature vector set, which is then fed into the recurrent network.

[0060] A recurrent network, including fully connected (FC) layers, unidirectional gated recurrent units (GRUs), and a softmax function, generates the improved speech signal one sample at a time. Since each output feature vector generally corresponds to a frame of multiple samples, the same output feature vector is used in generating the corresponding sample of the improved speech signal. For each current sample of the improved speech signal, the output feature vector and the previous samples already generated are concatenated before passing through a block of trained transforms terminating with a softmax activation to calculate the probability of each possible value for the current sample, and then this probability is used to calculate the cross-entropy loss of the clean speech sample.

[0061] As described in the previous section, the feature vector set can be generated from a distorted speech signal, but the feature vector set is expected to characterize the clean speech signal. In some embodiments, to improve the accuracy of the resulting WaveRNN model, for each current sample of the improved speech signal, the output feature vector and the previous sample of the known clean speech signal corresponding to the previous sample of the improved speech signal are concatenated before passing through a block of learned transforms that ends with a softmax activation (hence, for example, flow from 332 to 350) to calculate the probability of each possible value for the current sample. This probability is then used to calculate the cross-entropy loss of the clean speech sample and update the model parameter values.

[0062] In some embodiments, the original speech signal has a sampling rate of 48 kHz, a feature vector set is generated by a feature extraction model from down-sampled data with a sampling rate of 16 kHz, and a speech enhancement network generates an improved speech signal, again with a sampling rate of 48 kHz.

[0063] In some embodiments, the server 102 constructs an alternative speech enhancement model instead of the WaveRNN model. To utilize the second training set, the alternative speech enhancement model is also a generative model for restoring a clean speech signal. Examples of the alternative speech enhancement model include LPCNet, WaveNet, or SampleRNN.

[0064] In some embodiments, instead of training the feature extraction model and the speech enhancement model separately, the server 102 trains the two models together. Then, instead of potentially using separate datasets such as 306 and 336 to train the two models, the output 308 of the feature extraction model 320 can be fed directly into the generative deep model 350 during training, and the parameters of both models can be tuned during the same training process. In this way, the extracted features can be further refined to better suit the application of interest, in this case restoring a clean speech signal.

[0065] 4.3. Running the Deep Learning Model

[0066] In some embodiments, given a new corrupted coded speech signal, the server 102 generates a feature vector set using the PASE+ model. If the new speech signal is at a sampling rate (e.g., 48 kHz) that is common in today's multimedia communications, the server 102 downsamples the new speech signal to a lower sampling rate, e.g., 16 kHz, which is used to train the feature extraction model before running it. The server 102 then generates an improved signal from the feature vector set using the WaveRNN model. In this case, the samples used in generating the current sample of the improved speech signal are the previous samples generated for the improved speech signal. The generation process is autoregressive by randomly sampling from a predicted distribution of the current sample based on the previously generated samples.

[0067] In some embodiments, as described above, for a given new contaminated coded speech signal, the PASE+ model can be replaced with an alternative feature extraction model or the WaveRNN model can be replaced with an alternative speech enhancement model in generating an improved speech signal.

[0068] In some embodiments, the server 102 transmits the enhanced audio signal to an output device, such as an audio player or other computing device, which can further process the enhanced audio signal.

[0069] 5. Exemplary Processing

[0070] FIG. 4 illustrates an exemplary process performed by a voice management computer system according to some embodiments disclosed herein. FIG. 4 is illustrated in a simplified schematic form to provide a clear example, and other embodiments may include more, fewer, or different elements combined in various ways. Each of FIG. 4 is intended to disclose an algorithm, plan, or outline that, when executed, can be used to implement one or more computer programs or other software elements that perform the functional improvements and technical advances described herein. Moreover, the flow diagrams herein are described at the same level of detail that those skilled in the art would typically use to communicate to one another about the algorithms, plans, or specifications that form the basis of the software programs that they plan to code or implement using their accumulated skill and knowledge.

[0071] At step 402, the server 102 is programmed to obtain coded audio data including a first set of frames.

[0072] In some embodiments, the server 102 is programmed to receive the original coded data and downsample the original coded data to obtain the coded audio data. The original coded data may correspond to a sampling rate of 48 kHz and the coded audio data may correspond to a sampling rate of 16 kHz. The coded audio data may include noise or reverb.

[0073] In step 404, the server 102 is programmed to extract a set of feature vectors from the coded audio data using a self-supervised deep learning model including a neural network, the set of feature vectors being each extracted from the first set of frames.

[0074] In some embodiments, the self-supervised deep learning model includes an encoder and multiple workers, each of which performs a self-supervised task associated with a distinct speech feature, and one of the multiple workers can perform a self-supervised task associated with a predefined sampling strategy that draws anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

[0075] In step 406, the server 102 is programmed to generate enhanced speech data including a second set of frames from the set of feature vectors using a generative deep learning model including a neural network, the enhanced speech data corresponding to clean speech in the coded audio data.

[0076] In some embodiments, the generative deep learning model includes a conditional network and a recurrent network. The conditional network transforms the feature vector set into an output feature vector set by considering multiple frames each time. The recurrent network generates enhanced speech data from the output feature vector set, one sample at a time, where each frame of the second set of frames includes multiple samples. The recurrent network can generate new samples for each frame of enhanced speech data using corresponding feature vectors from the feature vector set and previously generated samples of the enhanced speech data.

[0077] In some embodiments, the server 102 is programmed to obtain a training set of distorted, downsampled audio signals for a given sampling rate, and the server is further configured to build a self-supervised deep learning model using the training set of distorted, downsampled audio signals.

[0078] In some embodiments, the server 102 is programmed to obtain a second data set of a clean audio signal at a predetermined sampling rate corresponding to the coded and downsampled audio signal. The server is also configured to distort a copy of the second data set with one or more artifacts caused by the recording environment, the recording device, or the coding algorithm to obtain the first training set. The server is further configured to build a self-supervised deep learning model also using the second data set.

[0079] In some embodiments, the server 102 is programmed to obtain a dataset of coded, downsampled audio signals for a predefined sampling rate. The server 102 is further configured to generate a training set of feature vector sets from the dataset using a self-supervised deep learning model. The server 102 is further configured to build a generative deep learning model using the training set of feature vector sets.

[0080] In some embodiments, the server 102 is programmed to obtain a third data set of a clean audio signal at a predetermined sampling rate corresponding to the coded downsampled audio signal, and the server is further configured to build a generative deep learning model using the third data set as well.

[0081] In some embodiments, the server 102 is programmed to obtain a dataset of distorted audio signals downsampled to a predetermined sampling rate and to train a combined model that includes a self-supervised deep learning model connected to a generative deep learning model using the dataset.

[0082] Various aspects of the disclosed embodiments may be apparent from the enumerated example embodiments (EEE) set forth below.

[0083] (EEE1) A computer-implemented method for recovering clean speech from coded audio data, comprising: obtaining coded audio data comprising a first set of frames; extracting a set of feature vectors from the coded audio data using a self-supervised deep learning model including a neural network, the set of feature vectors being each extracted from the first set of frames; generating enhanced speech data comprising a second set of frames from the set of feature vectors using a generative deep learning model comprising a neural network, the enhanced speech data corresponding to clean speech in the coded audio data; The method includes:

[0084] (EEE2) The computer-implemented method of claim 1, further comprising receiving original coded data, wherein the obtaining step comprises downsampling the original coded data.

[0085] (EEE3) The computer-implemented method of claim 2, wherein the original coded data corresponds to a sampling rate of 48 kHz and the coded audio data corresponds to a sampling rate of 16 kHz.

[0086] (EEE4) A computer-implemented method according to any one of claims 1 to 3, wherein the coded audio data includes noise or reverb.

[0087] (EEE5) The computer-implemented method of any one of claims 1 to 4, wherein the self-supervised deep learning model includes an encoder and a plurality of workers, each of which performs a self-supervised task associated with a distinct speech feature, and a worker of the plurality of workers performs a self-supervised task associated with a predefined sampling strategy that draws anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

[0088] (EEE6) The computer-implemented method of any one of claims 1 to 5, wherein the generative deep learning model includes a conditional network and a recurrent network, the conditional network transforms the feature vector set into an output feature vector set by considering multiple frames each time, and the recurrent network generates the enhanced speech data from the output feature vector set one sample at a time, and each frame of the second set of frames includes multiple samples.

[0089] (EEE7) obtaining a training set of distorted speech signals at a specific sampling rate lower than a predetermined sampling rate; building the self-supervised deep learning model using a training set of the distorted speech signals; 7. The computer-implemented method of claim 1, further comprising:

[0090] (EEE8) obtaining a data set of downsampled audio signals for a given sampling rate; generating a training set of feature vectors from the dataset using the self-supervised deep learning model; constructing the generative deep learning model using a training set of the set of feature vectors; 8. The computer-implemented method of claim 1, further comprising:

[0091] (EEE9) obtaining a data set of distorted audio signals at a specific sampling rate lower than a predetermined sampling rate; training a combined model that includes the self-supervised deep learning model connected to the generative deep learning model using the dataset; 9. The computer-implemented method of claim 1, further comprising:

[0092] (EEE10) A system for recovering clean speech from coded audio data, comprising: Memory, one or more processors coupled to the memory and configured to execute the computer-implemented method of any one of claims 1 to 9; A system including:

[0093] (EEE11) A computer-readable non-transitory storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed, performing a method for recovering clean speech from coded audio data, the method comprising: obtaining a data set of a coded audio signal downsampled to a predetermined sampling rate; generating a training set of feature vectors from the dataset using the self-supervised deep learning model; building a generative deep learning model using a training set of the set of feature vectors; extracting a set of feature vectors from the coded audio data using the self-supervised deep learning model; generating enhanced speech data from the set of feature vectors using the generative deep learning model; 1. A computer-readable non-transitory storage medium comprising:

[0094] (EEE12) The method further comprises: obtaining a first training set of coded speech signals at a particular sampling rate lower than the predetermined sampling rate; generating the self-supervised deep learning model from the first training set; 12. The computer-readable non-transitory storage medium of claim 11, further comprising:

[0095] (EEE13) The method further comprises the steps of: obtaining a second data set of a clean speech signal at a particular sampling rate corresponding to the coded speech signal; distorting a copy of the second data set with one or more artifacts caused by the recording environment, the recording device, or the coding algorithm to obtain the first training set; Further comprising: The computer-readable, non-transitory storage medium of claim 12 , wherein the generating step is performed further using the second data set.

[0096] (EEE14) The method further comprises the step of receiving original coded data at the predetermined sampling rate, 14. The computer-readable non-transitory storage medium of claim 11, wherein the extracting step includes a step of downsampling the original coded data.

[0097] (EEE15) A computer-readable non-transitory storage medium according to any one of claims 11 to 14, wherein the data set of the downsampled coded audio signal includes noise or reverb, and the coded audio data also includes noise or reverb.

[0098] (EEE16) The self-supervised deep learning model includes an encoder and a plurality of workers, each of which performs a self-supervised task associated with a distinct speech feature, and each of which performs a self-supervised task associated with a predefined sampling strategy that draws anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder. A computer-readable non-transitory storage medium as described in any of claims 11 to 15.

[0099] (EEE17) The computer-readable non-transitory storage medium of any one of claims 11 to 16, wherein the coded speech data includes a first set of frames, the set of feature vectors are each extracted from the first set of frames, and the enhanced speech data includes a second set of frames.

[0100] (EEE18) The computer-readable non-transitory storage medium of claim 17, wherein the generative deep learning model includes a conditional network and a recurrent network, the conditional network transforming a set of feature vectors among the set of feature vectors into an output feature vector set by considering multiple frames each time, and the recurrent network generating the enhanced speech data from the output feature vector set one sample at a time, and each frame of the second set of frames including multiple samples.

[0101] (EEE19) The computer-readable non-transitory storage medium of claim 18, wherein the recurrent network generates new samples for each frame of the enhanced speech data using corresponding feature vectors from the set of feature vectors and previously generated samples of the enhanced speech data.

[0102] (EEE20) The method further comprises: obtaining a second data set of a clean speech signal at the predetermined sampling rate corresponding to the downsampled coded speech signal; 20. The computer-readable, non-transitory storage medium of claim 11, wherein the constructing step is further performed using the second data set.

[0103] 6. Hardware Implementation

[0104] According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and / or other computing devices connected using a network, such as a packet data network. The computing device may be hard-wired to perform the techniques, or may include digital electronic devices such as at least one application specific integrated circuit (ASIC) or field programmable gate array (FPGA) that are permanently programmed to perform the techniques, or may include at least one general-purpose hardware processor that is programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such a computing device may incorporate hard-wired logic, an ASIC, or an FPGA with custom programs to accomplish the described techniques. The computing device may be a server computer, a workstation, a personal computer, a portable computer system, a handheld device, a mobile computing device, a wearable device, a body-worn or implantable device, a smartphone, a smart appliance, an internetworking device, an autonomous or semi-autonomous device such as a robot or an unmanned ground or aerial vehicle, any other electronic device that incorporates hard-wired and / or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0105] Figure 5 is a block diagram illustrating an exemplary computer system on which embodiments may be implemented. In the example of Figure 5, computer system 500 and instructions for implementing the disclosed techniques in hardware, software, or a combination of hardware and software are illustrated generally, e.g., as boxes and circles, at the same level of detail commonly used by those skilled in the art to which this disclosure pertains for computer architecture and communications related to computer system implementations.

[0106] Computer system 500 includes an input / output (I / O) subsystem 502, which may include a bus and / or other communication mechanisms for communicating information and / or instructions between components of computer system 500 via electronic signal paths. I / O subsystem 502 may include an I / O controller, a memory controller, and at least one I / O port. The electronic signal paths are depicted in the figures diagrammatically, for example as lines, single-headed arrows, or double-headed arrows.

[0107] At least one hardware processor 504 is connected to the I / O subsystem 502 for processing information and instructions. The hardware processor 504 may include, for example, a general purpose microprocessor or microcontroller and / or an embedded system or a special purpose microprocessor such as a graphic processing unit (GPU) or a digital signal processor or an ARM processor. The processor 504 may include an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

[0108] The computer system 500 includes one or more units of memory 506, such as a main memory, which is connected to the I / O subsystem 502 for electronically and digitally storing data and instructions to be executed by the processor 504. The memory 506 may include volatile memory, such as various forms of random access memory (RAM) or other dynamic storage devices. The memory 506 may be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 504. Such instructions, when stored on a non-transitory computer-readable recording medium accessible to the processor 504, can make the computer system 500 into a special-purpose machine customized to perform the operations specified in the instructions.

[0109] The computer system 500 further includes a non-volatile memory such as a read only memory (ROM) 508 or other static storage device connected to the I / O subsystem 502 for storing information and instructions for the processor 504. The ROM 508 may include various types of programmable ROM (PROM) such as erasable programmable ROM (EPROM) or electrically erasable programmable ROM (EEPROM). A unit of permanent storage 510 may include various types of non-volatile RAM (NVRAM) such as FLASH memory or solid state storage, magnetic disks, or optical disks such as CD-ROM or DVD-ROM, may be connected to the I / O subsystem 502 for storing information and instructions. The storage device 510 is an example of a non-volatile computer readable medium that may be used to store instructions and data that, when executed by the processor 504, cause the computer-implemented methods to perform the techniques herein.

[0110] The instructions in the memory 506, ROM 508, or storage device 510 may include one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps. The instructions may include an operating system and / or system software; one or more libraries supporting multimedia, programming, or other functionality; data protocol instructions or stacks for implementing TCP / IP, HTTP, or other communication protocols; file processing instructions for interpreting and rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a graphic user interface (GUI), a command line interface, or a text user interface; application software, such as an office suite, Internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or miscellaneous applications. The instructions may implement a web server, a web application server, or a web client. The instructions may be organized as a presentation layer, an application layer, and a data storage layer, such as a relational database system using Structured Query Language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0111] The computer system 500 may be connected to at least one output device 512 via the I / O subsystem 502. In one embodiment, the output device 512 is a digital computer display. Examples of displays that may be used in various embodiments include a touch screen display or a light emitting diode (LED) display or a liquid crystal display (LCD) or an e-paper display. The computer system 500 may include other types of output devices 512 as an alternative or in addition to a display device. Examples of other types of output devices 512 include a printer, a ticket printer, a plotter, a projector, an audio or video card, a speaker, a buzzer, or a piezoelectric or other audible device, a lamp or LED or LCD indicator, a haptic device, an actuator or servo.

[0112] At least one input device 514 is connected to the I / O subsystem 502 for communicating signals, data, command selections, or gestures to the processor 504. Examples of input device 514 include a touch screen, a microphone, a still or video digital camera, alphanumeric and other keys, a keypad, a keyboard, a graphics tablet, an image scanner, a joystick, a clock, switches, buttons, dials, slides, and / or various types of sensors such as force sensors, motion sensors, thermal sensors, accelerometers, gyroscopes, inertial measurement unit (IMU) sensors, and / or various types of transceivers such as wireless, such as cellular or Wi-Fi, radio frequency (RF) or infrared (IR) transceivers, and global positioning system (GPS) transceivers.

[0113] Another type of input device is a control device 516 that may perform cursor control or other child-controlled functions, such as navigation within a graphical interface on a display screen, instead of or in addition to input functions. The control device 516 may be a touchpad, mouse, trackball, or cursor direction keys for communicating directional information and command selections to the processor 504 and for controlling cursor movement on the display 512. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to specify a position in a plane. Another type of input device is a wired, wireless, or optical control device, such as a joystick, wand, console, steering wheel, pedals, gear shift mechanism, or other type of control device. The input device 514 may include a combination of multiple different input devices, such as a video camera and a depth sensor.

[0114] In another embodiment, computer system 500 may include an Internet of Things (IoT) device in which one or more of output device(s) 512, input device(s) 514, and controller(s) 516 are omitted. Alternatively, in such an embodiment, input device(s) 514 may include one or more cameras, motion detectors, thermometers, microphones, earthquake detectors, other sensors or detectors, measurement devices, or encoders, and output device(s) 512 may include a dedicated display, such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator, or a servo.

[0115] When computer system 500 is a mobile computing device, input device 514 may include a Global Positioning System (GPS) receiver connected to a GPS module capable of triangulating against multiple GPS satellites to determine and generate location data such as a geographic position or latitude and longitude values ​​of the geophysical location of computer system 500. Output device 512 may include hardware, software, firmware, and interfaces for generating position report packets, notifications, pulse or heartbeat signals, or other circular data transmissions that, alone or in combination with other application specific data, specify the location of computer system 500 directed to host 524 or server 530.

[0116] The computer system 500 may implement the techniques described herein using customized hard-wired logic, at least one ASIC or FPGA, firmware and / or program instructions or logic that, when loaded and used or executed in conjunction with the computer system, cause or program the computer system to operate as a special-purpose machine. According to one embodiment, the techniques described herein may be performed by the computer system 500 in response to the processor 504 executing at least one sequence of at least one instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as the storage device 510. Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0117] The term "storage media" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific fashion. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as memory 506. Common types of storage media may include, for example, hard disks, solid state drives, flash drives, magnetic data storage media, any optical or physical data storage media, memory chips, etc.

[0118] Storage media is distinct from, but may be used in the context of, transmission media, which are involved in transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a bus in I / O subsystem 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0119] Various types of media may be involved in carrying at least one sequence of at least one instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions using a modem over a communications link, such as a fiber optic or coaxial cable or a telephone line. A modem or router local to the computer system 500 can receive the data over the communications link and convert the data for reading by the computer system 500. For example, a receiver, such as a radio frequency antenna or infrared detector, can receive the data carried in a radio or optical signal and appropriate circuitry can provide the data to the I / O subsystem 502, for example placing the data on a bus. The I / O subsystem 502 carries the data to memory 506, from which the processor 504h reads and executes the instructions. The instructions received by memory 506 may optionally be stored on storage device 510 either before or after execution by the processor 504.

[0120] Computer system 500 also includes a communication interface 518 connected to bus 502. Communication interface 518 provides a two-way data communication coupling to a network link 520 that is directly or indirectly connected to at least one communication network, such as a network 522 or a public or private cloud on the Internet. For example, communication interface 518 may be an Ethernet network interface, an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of communication line, such as an Ethernet cable or any type of metallic or fiber optic line or a telephone line. Network 522 broadly represents a local area network (LAN), a wide area network (WAN), a campus network, an internetwork, or any combination thereof. Communication interface 518 may include a LAN card to provide a data communication connection to a compatible LAN, or a cellular radiotelephone interface wired to transmit or receive cellular data according to a cellular radiotelephone radio networking standard, or a satellite radio interface wired to transmit or receive digital data according to a satellite radio networking standard. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic or optical signals via signal paths that carry digital data streams representing various types of information.

[0121] Network link 520 typically provides electrical, electromagnetic or optical data communication directly or through at least one network to other data devices, for example using satellite, cellular, Wi-Fi, or BLUETOOTH technology. For example, network link 520 may provide a connection through network 522 to a host computer 524.

[0122] Further, network link 520 may provide connectivity through network 522 or through internetworking devices and / or computers to other computing devices operated by an Internet Service Provider (ISP) 526. ISP 526 provides data communication services through a worldwide packet data communications network represented as Internet 528. A server computer 530 may be connected to Internet 528. Server 530 broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or a computer running a containerized program system such as DOCKER or KUBERNETES. Server 530 may represent an electronic digital service implemented using multiple computers or instances and accessed and used by sending Web service requests, Uniform Resource Locator (URL) strings with parameters of HTTP payloads, Application Programming Interface (API) calls, app service calls, or other service calls. Computer system 500 and server 530 may form elements of a distributed computing system including other computers, processing clusters, server farms, or other organizations of computers that cooperate to perform tasks or execute applications or services. The server 530 may include one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps.The instructions may include operating system and / or system software; one or more libraries supporting multimedia, programming, or other functionality; data protocol instructions or stacks for implementing TCP / IP, HTTP, or other communications protocols; file formatting instructions for interpreting or rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a graphic user interface (GUI), a command line interface, or a text user interface; application software such as an office suite, Internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or miscellaneous applications. The server 530 may include a web application server hosting a presentation layer, an application layer, and a data storage layer such as a relational database system using Structured Query Language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0123] Computer system 500 can send messages and receive data and instructions, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a necessary code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518. The received code may be executed by processor 504 once received and / or stored in storage device 510, or other non-volatile storage for later execution.

[0124] Execution of the instructions described in this chapter implements a process in the form of an instance of a running computer program, which may consist of program code and its current activity. Depending on the operating system (OS), a process may consist of multiple threads of execution that execute instructions simultaneously. In this context, a computer program is a passive collection of instructions, while a process may be the actual execution of those instructions. Multiple processes may be associated with the same program, e.g., opening multiple instances of the same program often means that multiple processes are running. Multitasking may be implemented to allow multiple processes to share the processor 504. While each processor 504 or processor core executes a single task at a time, the computer system 500 may be programmed to implement multitasking so that each processor can switch between running tasks without waiting for each task to finish. In an embodiment, the switch may be performed when a task performs an input / output operation, when a task indicates that it is available to switch, or due to a hardware interrupt. Time sharing may be implemented to perform context switches quickly and provide the appearance of multiple processes running simultaneously, thereby speeding up the response of interactive user applications. In embodiments, for security and reliability, the operating system prevents direct communication between independent processes and provides a strictly mediated and controlled inter-process communication facility.

[0125] 7. Extensions and Substitutions

[0126] In the foregoing specification, embodiments of the present disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings should therefore be considered in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the present disclosure, and what the applicant intends to be the scope of the present disclosure, is the literal equivalent of the claims issued in a particular form, including any later amendments from this application.

Claims

1. 1. A computer-implemented method for recovering clean speech from coded audio data, comprising the steps of: obtaining coded audio data comprising a first set of frames; extracting a set of feature vectors from the coded audio data using a self-supervised deep learning model including a neural network, the set of feature vectors being each extracted from the first set of frames; generating enhanced speech data comprising a second set of frames from the set of feature vectors using a generative deep learning model comprising a neural network, the enhanced speech data corresponding to clean speech in the coded audio data; 23. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , further comprising receiving original coded data, said obtaining comprising downsampling the original coded data.

3. 3. The computer-implemented method of claim 2, wherein the original coded data corresponds to a sampling rate of 48 kHz and the coded audio data corresponds to a sampling rate of 16 kHz.

4. The computer-implemented method of claim 1 , wherein the coded audio data includes noise or reverb.

5. the self-supervised deep learning model includes an encoder and a plurality of workers, each worker of the plurality of workers performing a self-supervised task associated with a distinct speech feature; 2. The computer-implemented method of claim 1 , wherein a worker of the plurality of workers performs a self-supervised task associated with a predefined sampling strategy that draws anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

6. The generative deep learning model includes a conditional network and a recurrent network; the conditional network transforms the set of feature vectors into a set of output feature vectors by considering multiple frames each time; 2. The computer-implemented method of claim 1, wherein the recurrent network generates the enhanced speech data one sample at a time from the set of output feature vectors, and each frame of the second set of frames includes multiple samples.

7. obtaining a training set of distorted speech signals at a particular sampling rate that is lower than a predetermined sampling rate; building the self-supervised deep learning model using a training set of the distorted speech signals; The computer-implemented method of claim 1 further comprising:

8. obtaining a data set of a downsampled audio signal for a given sampling rate; generating a training set of feature vectors from the dataset using the self-supervised deep learning model; constructing the generative deep learning model using a training set of the set of feature vectors; The computer-implemented method of claim 1 further comprising:

9. obtaining a data set of a distorted audio signal at a particular sampling rate that is lower than a predetermined sampling rate; training a combined model that includes the self-supervised deep learning model connected to the generative deep learning model using the dataset; The computer-implemented method of claim 1 further comprising:

10. 1. A system for recovering clean speech from coded audio data, comprising: Memory, one or more processors coupled to the memory and configured to execute the computer-implemented method of any one of claims 1 to 9; A system including:

11. 1. A computer-readable non-transitory storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed, performing a method for recovering clean speech from coded audio data, the method comprising: obtaining a data set of a coded audio signal downsampled to a predetermined sampling rate; generating a training set of feature vectors from the dataset using a self-supervised deep learning model; building a generative deep learning model using a training set of the set of feature vectors; extracting a set of feature vectors from the coded audio data using the self-supervised deep learning model; generating enhanced speech data from the set of feature vectors using the generative deep learning model; 1. A computer-readable non-transitory storage medium comprising:

12. The method comprises: obtaining a first training set of coded speech signals at a particular sampling rate lower than the predetermined sampling rate; generating the self-supervised deep learning model from the first training set; The computer-readable non-transitory storage medium of claim 11 , further comprising:

13. The method comprises: obtaining a second data set of a clean speech signal at a particular sampling rate corresponding to the coded speech signal; distorting a copy of the second data set with one or more artifacts caused by the recording environment, the recording device, or the coding algorithm to obtain the first training set; Further comprising: The computer-readable, non-transitory storage medium of claim 12 , wherein the generating step is performed further using the second data set.

14. The method further comprises receiving original coded data at the predetermined sampling rate; The computer-readable non-transitory storage medium of claim 11 , wherein the extracting step comprises downsampling the original coded data.

15. 12. The computer-readable non-transitory storage medium of claim 11, wherein the downsampled coded audio signal data set includes noise or reverb, and the coded audio data also includes noise or reverb.

16. The self-supervised deep learning model includes an encoder and a plurality of workers; each worker of the plurality of workers performs a self-supervised task associated with a distinct speech feature; 12. The computer-readable non-transitory storage medium of claim 11, wherein a worker of the plurality of workers performs a self-supervised task associated with a predefined sampling strategy that draws anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

17. the coded audio data includes a first set of frames; the set of feature vectors are each extracted from the first set of frames; The computer-readable, non-transitory storage medium of claim 11 , wherein the enhanced audio data comprises a second set of frames.

18. The generative deep learning model includes a conditional network and a recurrent network; the conditional network transforms a set of feature vectors from the set of feature vectors into an output set of feature vectors by considering a number of frames each time; 20. The computer-readable, non-transitory storage medium of claim 17, wherein the recurrent network generates the enhanced speech data from a set of output feature vectors one sample at a time, and each frame of the second set of frames includes multiple samples.

19. 20. The computer-readable, non-transitory storage medium of claim 18, wherein the recurrent network generates new samples for each frame of the enhanced speech data using corresponding feature vectors from the set of feature vectors and previously generated samples of the enhanced speech data.

20. The method comprises: obtaining a second data set of a clean speech signal at the predetermined sampling rate corresponding to the downsampled coded speech signal; The computer-readable non-transitory storage medium according to any one of claims 11 to 19, wherein the constructing step is further performed using the second data set.

Images