Speech enhancement using deep generative models

Abstract

This invention provides a system and method for generating enhanced speech data using robust speech features. [Solution] The method implemented by the system includes the steps of: acquiring coded audio data; generating a set of feature vectors from the coded audio data, including the polluted speech, using a self-supervised deep learning model; and generating improved audio data from the set of feature vectors, corresponding to clean speech, using a generative deep learning model.

Description

Technical Field

[0001] [Related Applications] This application claims priority to Spanish Patent Application No. P202230148 filed on February 23, 2022, and US Provisional Application No. 63 / 431,590 filed on December 9, 2022, which are hereby incorporated by reference in their entirety.

[0002] [Technical Field] This application relates to speech processing and machine learning.

Background Art

[0003] The approaches described in this chapter are approaches that can be pursued, but are not necessarily approaches that have been previously devised or pursued. Therefore, unless otherwise indicated, none of the approaches described in this chapter should be regarded as prior art solely by virtue of being included in this chapter.

[0004] Speech coding has found a primary use in secure communications and later enabled low-cost mobile and Internet communications. Due to the continuously decreasing bitrate of speech coders, the decoded speech is troubled by various quality problems. It would be useful to address such quality problems.

Summary of the Invention

[0005] A computer method for restoring clean speech from coded audio data. The method comprises obtaining coded audio data, which includes a first frameset. The method further comprises extracting a set of feature vectors from the coded audio data using a self-supervised deep learning model, which includes a neural network, the set of feature vectors being extracted from each of the first framesets. The method further comprises generating enhanced speech data, which includes a second frameset, from the set of feature vectors using a generative deep learning model, which includes a neural network, the enhanced speech data corresponding to the clean speech in the coded audio data.

[0006] The techniques described herein offer advantages over conventional audio processing techniques. For example, this method enables the generation of improved speech data from actual coded audio data, often containing contaminated speech. Generation is based on robust features derived from self-supervised learning of speech characteristics at various levels, from low-level raw audio characteristics to high-level speaker-related characteristics, which helps achieve high quality in the improved speech data despite distortion and coding. The improved speech data is efficiently enhanced using a relatively compact deep learning model that can run in real time as the raw speech signal is generated. The improved speech data results in better perception of the audio and better user enjoyment of the audio. [Brief explanation of the drawing]

[0007] Exemplary embodiments of the present invention are described by example, not limiting them, and similar reference numerals in the accompanying figures represent similar elements.

[0008] [Figure 1] This document illustrates an exemplary network computer system that can implement various embodiments.

[0009] [Figure 2]The following are illustrative components of the voice management computer system according to the disclosed embodiment.

[0010] [Figure 3] This shows the training process for both the feature extraction model and the speech enhancement model.

[0011] [Figure 4] The following illustrates exemplary processes performed by voice management computer systems according to some embodiments disclosed herein.

[0012] [Figure 5] Block diagram showing a computer system in which one embodiment of one model can be implemented. [Modes for carrying out the invention]

[0013] In the following detailed description, numerous specific details are described for illustrative purposes to provide a complete understanding of exemplary embodiments of the invention. However, it is evident that exemplary embodiments may be carried out without these specific details. In other examples, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the exemplary embodiments.

[0014] Embodiments are described in the following sections according to the following outline. 1. Overview 2. Exemplary Computing Environment 3. Exemplary Computer Components 4. Functional Description 4.1. Training the Feature Extraction Model 4.1.1. Data Distortion 4.1.2. Data Downsampling 4.1.3. Feature Extraction Model 4.2. Training of the voice enhancement model 4.2.1. Data Collection 4.2.2. Voice Enhancement Model 4.3. Execution of Deep Learning Model 5. Exemplary Processing 6. Hardware Implementation 7. Extension and Alternatives

[0015] 1. Overview

[0016] A system is disclosed that generates enhanced audio data using robust audio features. In some embodiments, the system is programmed to generate a set of feature vectors from a given audio data coded with contaminated audio using a self-supervised deep learning model. The system is further programmed to generate improved audio data corresponding to clean audio from the set of feature vectors using a generative deep learning model.

[0017] In some embodiments, the system is programmed to train a self-supervised deep learning model. The training includes calculating the weights of an encoder using specific audio features of clean audio signals, where the encoder is configured to generate a set of feature vectors from contaminated coded audio signals. The audio features can include low-level audio features such as energy or pitch, or high-level audio features such as speaker accent. The audio signal represents a digitized audio waveform containing audio. As a result of digitization, one or more frames are generated, each frame containing one or more samples corresponding to a specific sampling rate. Distortion can occur due to extra audio or non-audio in the environment such as noise or reverberation, or limitations of recording equipment such as clipping. Coding is brought about by the application of a compression algorithm with a target bitrate. The trained self-supervised deep learning model is expected to generate a set of robust feature vectors that characterize the corresponding frame of clean audio signal for each frame of the contaminated coded audio signal.

[0018] In some embodiments, the clean audio signal is at a particular sampling rate, but the system is programmed to downsample the coded audio signal derivable from the clean audio signal to form a training dataset for a self-supervised deep learning model. For example, an audio signal at a sampling rate of 48 kHz can be downsampled to an audio signal at a sampling rate of 16 kHz. The downsampled audio signal is expected to continue to capture the major audio features that can be used to reconstruct the 48 kHz clean signal.

[0019] In some embodiments, the system is programmed to train a generative deep learning model. The training includes using the clean audio signal to calculate the weights of a recurrent neural network, which is configured to generate an improved audio signal from a set of feature vectors generated by a self-supervised deep learning model from the contaminated coded audio signal. The trained generative deep learning model is expected to generate an improved audio signal at a predetermined sampling rate, such as 48 kHz, even when the set of feature vectors is generated from an audio signal at a lower sampling rate, such as 16 kHz.

[0020] In some embodiments, when a new audio signal coded using any coding algorithm for any bitrate is provided, the system is configured to run the trained self-supervised deep learning model on the new audio signal to generate a set of feature vectors. The system is further configured to run the trained generative deep learning model on the set of feature vectors to generate an improved audio signal corresponding to the decoded clean audio signal.

[0021] This system offers technical advantages. As the bitrate of audio codecs continues to decline, decoded audio suffers from bandwidth loss, spectral holes, or other quality issues. This system addresses the technical problem of unpredictable quality of decoded audio signals resulting from the decline in the bitrate of the coding process. The system enables the generation of improved audio data from actual coded audio data, which often contains contaminated speech. Generation is based on robust features derived from self-supervised learning of speech characteristics at various levels, from low-level raw audio characteristics to high-level speaker-related characteristics, which helps achieve high quality in the improved audio data despite distortion and coding. The improved audio data is efficiently enhanced using a relatively compact deep learning model that can run in real time as the raw audio signal is generated. The improved audio data results in better perception of audio and better user enjoyment of audio.

[0022] 2. Exemplary Computing Environment

[0023] Figure 1 shows an exemplary network computer system in which various embodiments can be implemented. Figure 1 is shown in a simplified schematic form for clarity, and other embodiments may include more, fewer, or different elements.

[0024] In some embodiments, the networked computer system includes an audio management server computer 102 ("server"), one or more sensors 104 or input devices, and one or more output devices 110, which are connected in a communicative manner via direct physical connections or via one or more networks 118.

[0025] In some embodiments, Server 102 broadly represents one or more computers, virtual computing instances, and / or instances of applications programmed or configured with data structures and / or database records configured to host or perform functions related to restoring clean audio signals from corrupted coded audio signals. Server 102 may include server farms, cloud computing platforms, parallel computers, or any other computing equipment having sufficient computing power for data processing, data storage, and network communication for the above functions.

[0026] In some embodiments, each of the one or more sensors 104 may include a microphone or another digital recording device that converts sound into electrical signals. Each sensor is configured to transmit raw or processed audio data to the server 102. Each sensor may include a processor or be integrated into a typical client device such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device, and the processor coupled to the sensor may perform initial processing of the audio data.

[0027] In some embodiments, each of the one or more output devices 110 may include a speaker or another digital playback device that converts the electrical signal back into sound. Each output device is programmed to play back audio data received from the server 102. Similar to sensors, output devices may include a processor or be integrated into a typical client device such as a desktop computer, laptop computer, tablet computer, smartphone, or wearable device, where the processor coupled to the sensor can perform subsequent processing of the audio data.

[0028] One or more networks 118 can be implemented by any medium or mechanism that provides data exchange between the various elements in Figure 1. Examples of networks 118 include, but are not limited to, one or more cellular networks, near-field communication (NFC) networks, local area networks (LANs), wide area networks (WANs), the Internet, terrestrial or satellite links, etc., which are communicatively coupled with data connectivity to computing devices via cellular antennas.

[0029] In some embodiments, the server 102 is programmed to receive input audio data corresponding to sounds in a given environment from one or more sensors 104. The input audio data may contain multiple frames over time. The server 102 is then programmed to process the coded input audio data, which typically has a mixture of speech and noise, to produce improved audio data corresponding to clean speech. The server may also be programmed to transmit output audio data to one or more output devices.

[0030] 3. Exemplary Computer Components

[0031] Figure 2 shows exemplary components of a voice management computer system according to the disclosed embodiment. The figure is illustrative only, and server 102 may include fewer or more functional or storage components. Each functional component may be implemented as a software component, a general-purpose or specific-purpose hardware component, a firmware component, or any combination thereof. Each functional component may also be coupled with one or more storage components (not shown). Storage components may be implemented using a relational database, an object database, a flat file system, or a Javascript Object Notation (JSON) store. Storage components may connect to functional components locally or over a network using program calls, remote procedure call (RPC) functionality, or a messaging bus. Components may be self-contained or not. Depending on implementation-specific or other considerations, components may be functionally or physically centralized or distributed.

[0032] In some embodiments, the server 102 includes a feature extraction model training instruction 202, a speech enhancement model training instruction 206, a model execution instruction 208, and a communication interface instruction 210. The server 102 also includes a database 220.

[0033] In some embodiments, the feature extraction model training instruction 202 enables the training of a deep learning model for extracting audio features corresponding to a clean audio signal from a coded, polluted audio signal. The deep learning model includes various neural networks or other self-supervised models that can learn features from a clean audio signal.

[0034] In some embodiments, the speech enhancement model training instruction 206 enables the training of a deep learning model to generate an improved speech signal from a set of audio features generated by a feature extraction model. The deep learning model includes various neural networks or other generative models that can generate an improved speech signal from audio features corresponding to a clean speech signal.

[0035] In some embodiments, the model execution instruction 208 enables the execution of a deep learning model for generating improved speech data from polluted coded speech data. The execution may include running a feature extraction model on coded speech data containing a mixture of speech and non-speech to generate a feature vector set. The execution may further include running a speech enhancement model on the feature vector set to generate improved speech data corresponding to clean speech.

[0036] In some embodiments, the communication interface instruction 210 enables communication with other systems or devices over a computer network. Communication may include receiving audio data or trained deep learning models from an audio source or other system. Communication may also include transmitting enhanced speech data to other processing or output devices.

[0037] In some embodiments, the database 220 is programmed or configured to manage the storage and access of related data, such as audio data including clean audio signals, polluted audio signals, coded audio signals, or downsampled audio signals, digital models, features extracted from the audio data, or the results of running digital models.

[0038] 4. Functional Description

[0039] Speech signals, i.e., digital audio data containing speech and typically represented as waveforms in the time domain, contain a rich set of acoustic and linguistic properties, ranging from individual lexical units such as phonemes or words to speaker characteristics such as the speaker's intent or emotional state. However, these properties of speech are not adequately captured by low-level signal processing characteristics such as the amplitude of the wave signal, the logarithmic Mel spectrogram, or the Mel frequency cepstrum coefficients (MFCCs). Codified speech has already lost information. Reconstructing the original speech using only low-level signal processing characteristics extracted from coded speech can be particularly difficult because it does not fully characterize the speech signal.

[0040] Therefore, in some embodiments, the server uses a feature extraction model, which is a self-supervised deep learning model, to identify or extract characteristics from coded speech. The extracted features are then used to construct a speech enhancement model, which is a generative deep learning model.

[0041] 4.1. Training the Feature Extraction Model

[0042] 4.1.1. Data Distortion

[0043] Audio signals are generally distorted by various forms of contamination caused by the environment or recording equipment, such as reverberation, added noise, overlapping voices, temporal or frequency masking, and clipping. Audio coding introduces yet another level of distortion.

[0044] Figure 3 shows the training process for the feature extraction model and the speech enhancement model. In some embodiments, server 102 expects a feature extraction model that is robust to speech distortion. Therefore, server 102 prepares a first dataset of speech signals, such as 306, distorted to varying degrees for the feature extraction model. The speech signals in the first dataset may contain contamination affecting different durations or frequency bands. An example of an approach that introduces such contamination into clean speech signals, such as 304, can be found in the paper by Ravanelli et al. titled "Multi-task self-supervised learning for Robust Speech Recognition," which concerns an improved version of a problem-agnostic speech encoder (PASE+). The speech signals in the first dataset may also follow different codecs, such as linear predictive coding (LPC) or modified discrete cosine transform (MDCT), with different bitrates, such as between 32kbps and 256kbps. For training purposes, the speech signals can be processed using an Advanced Audio Coding (AAC) codec with different bitrates. The bitrate ranges from 16kbps to 64kbps.

[0045] 4.1.2. Data Downsampling

[0046] A 48kHz sampling rate for digital audio is common in today's multimedia communications. Lower sampling rates can be used to train feature extraction models, based on the reasonable assumption that acoustic and linguistic content, such as phonemes and pitch, primarily resides in the low-frequency range. Using lower sampling rates reduces the amount of data required and improves the ease of finding the necessary data. Furthermore, this assumption includes the fact that the low-frequency range is generally below 8kHz.

[0047] Therefore, in some embodiments, server 102 selects a lower sampling rate, between 8kHz and 48kHz, for example, 16kHz. Server 102 then includes the 16kHz audio signal in the first dataset and uses the first dataset as the first training set. Alternatively, server 102 downsamples the 48kHz audio signal in the first dataset to obtain a first training set with a much larger, lower sampling rate than the first dataset. The feature extraction model is built primarily from the lower sampling rate training data, but the feature extraction model also works with features from the original higher sampling rate actual data, as will be further explained below, and is therefore used to extract them. Experiments demonstrate that a feature extraction model built from the first training set with a 16kHz sampling rate can reconstruct a clean 48kHz audio signal with high quality.

[0048] 4.1.3. Feature Extraction Model

[0049] In some embodiments, Server 102 constructs a feature extraction model such as 320 from a first training set. Server 102 can construct a PASE+ model, as described in the paper by Ravanelli et al. PASE+ uses a self-supervised learning approach to extract speech information at various levels, such as phonemes or speaker sentiment. PASE+ is a deep neural network that takes a distorted speech signal, such as a waveform, as input and generates as output a set of high-dimensional feature vectors that are expected to characterize the corresponding clean speech signal, with each feature being for one frame of the distorted speech signal. For example, each frame may be 10 ms long with 480 samples for a sampling rate of 48 kHz. Thus, Server 102 uses each distorted speech signal in a first training set to construct a PASE+ model and aims to generate a set of feature vectors such as 308 that characterize the corresponding clean, undistorted speech.

[0050] Specifically, PASE+ includes an encoder (including a quasi-recurrent neural network for learning long-term dependencies between time steps) that extracts groups of features as feature vectors from frames of distorted speech signals. PASE+ also includes numerous workers. Each worker is a small feedforward neural network that accomplishes a self-supervised signal transformation task corresponding to a known speech characteristic. For example, if the signal transformation task corresponds to an MFCC, the worker constructs a "target" MFCC value from a clean speech signal corresponding to the distorted speech signal (hence, e.g., a flow from 304 to 320) and determines the loss of the current feature vector from the target MFCC value. The signal transformation task can be considered self-supervised rather than supervised in that the classification or labeling of the ground truth (e.g., the clean speech signal) is not individually given, but the characteristics of the ground truth are automatically computed from the ground truth. The total loss of the current feature vector is calculated based on the losses determined by all the workers.

[0051] The experimental results show that introducing distortion into a clean audio signal increases the training data with greater diversity, and tends to further denoise the training data to learn distortion-invariant and robust features. Using the first training set of distorted signals, the worker is trained on an encoder to determine the encoder weights. As will be discussed further in Section 4.3, given a real audio signal, all that is needed is to run the trained encoder to obtain the corresponding group of robust features.

[0052] In some embodiments, the worker's task may relate to a filter bank, prosody (fundamental frequency, voiced / unvoiced probability, zero crossing rate, and energy interpolation logarithm), a logarithmic Mel spectrogram, or signal processing features such as MFCCs, or a sampling strategy used to capture speaker characteristics such as local information maximization or global information maximization, as discussed in Ravanelli et al.'s paper. In this way, PASE+ provides a deep and compact representation of various levels of speech abstraction, from low-level spectral information to high-level speaker information.

[0053] In some embodiments, server 102 constructs an alternative feature extraction model instead of the PASE+ model. To utilize the first training set, the alternative feature extraction model also employs a self-supervised or unsupervised learning approach to extract speech information at various levels. One example of an alternative feature extraction model is contrastive predictive coding (CPC), which uses a contrast loss function that depends on the coding of positive samples.

[0054] 4.2. Training of the voice enhancement model

[0055] 4.2.1. Data Collection

[0056] In some embodiments, server 102 starts with a sampling rate common in today's multimedia communications. Server 102 prepares a second dataset of audio signals, such as 334, that are contaminated to varying degrees for the speech enhancement model. The audio signals in the second dataset may also follow different codecs with different bitrates. Next, server 102 downsamples the audio signals in the second dataset to obtain a downsampled dataset with a lower sampling rate, such as 16kHz. Next, the server runs the downsampled dataset through a trained feature extraction model to obtain a second training set of feature vectors, such as 338, for the speech enhancement model.

[0057] 4.2.2. Voice Enhancement Model

[0058] In some embodiments, server 102 constructs an audio enhancement model, such as 350, from a second training set. Server 102 can construct a WaveRNN model, as described in the paper titled "Efficient Neural Audio Synthesis" by Kalchbrenner et al. WaveRNN efficiently generates high-quality samples using a sequential process. WaveRNN is a deep neural network that takes as input a set of feature vectors, one for each frame of the original audio signal, which is expected to characterize the corresponding clean audio signal, and generates as output a corresponding improved audio signal which is expected to correspond to the clean audio signal. Thus, server 102 can construct a WaveRNN model using the set of feature vectors generated from the distorted audio signal by the feature extraction model, with the goal of obtaining an undistorted improved audio signal, such as 340, for each distorted audio signal.

[0059] Specifically, WaveRNN includes a conditional network and a recurrent network. The conditional network consists of a pair of convolutional networks: a residual network and an upsampling network, operating in parallel. The residual network, whose dilation increases through blocks of learned transformations, can map a set of feature vectors to a latent representation, which is then split into multiple parts used as input to the subsequent recurrent network. Simultaneously, the set of feature vectors passes through the upsampling network to generate a second set of feature vectors that match the temporal size of the original audio signal. The outputs of these two convolutional networks are concatenated to form an output feature vector set, which is then fed into the recurrent network.

[0060] A recurrent network containing fully connected (FC) layers, unidirectional gated recurrent units (GRUs), and a softmax function generates an improved speech signal with one sample at a time. Since each output feature vector generally corresponds to a frame of multiple samples, the same output feature vector is used when generating the corresponding sample of the improved speech signal. For each current sample of the improved speech signal, the output feature vector and previously generated samples are concatenated before passing through a block of trained transformations ending with softmax activation, calculating the probability of each possible value for the current sample, and then using this probability to calculate the cross-entropy loss of the clean speech sample.

[0061] As described in the previous section, the feature vector set can be generated from a distorted audio signal, but the feature vector set is expected to characterize a clean audio signal. In some embodiments, to improve the accuracy of the resulting WaveRNN model, for each current sample of the improved audio signal, the output feature vector and a known previous sample of the clean audio signal corresponding to a previous sample of the improved audio signal are concatenated before passing through a block of trained transformations ending with softmax activation (thus, e.g., a flow from 332 to 350), and the probability of each possible value for the current sample is calculated. This probability is then used to calculate the cross-entropy loss of the clean audio sample and update the model parameter values.

[0062] In some embodiments, the original audio signal has a sampling rate of 48 kHz, the feature vector set is generated by a feature extraction model from downsampled data at a sampling rate of 16 kHz, and the audio enhancement network generates an improved audio signal again at a sampling rate of 48 kHz.

[0063] In some embodiments, server 102 constructs an alternative speech augmentation model instead of the WaveRNN model. To utilize a second training set, the alternative speech augmentation model is also a generative model for reconstructing a clean speech signal. Examples of alternative speech augmentation models include LPCNet, WaveNet, or SampleRNN.

[0064] In some embodiments, instead of training the feature extraction model and the speech enhancement model separately, the server 102 trains the two models together. Then, instead of potentially using separate datasets such as 306 and 336, the output 308 of the feature extraction model 320 can be directly fed into the generative deep model 350 during training to train the two models, and the parameters of both models can be adjusted during the same training process. In this way, the extracted features can be further refined and better adapted to the application of interest, in this case, restoring a clean speech signal.

[0065] 4.3. Running Deep Learning Models

[0066] In some embodiments, given a new, corrupted, coded audio signal, server 102 generates a feature vector set using a PASE+ model. If the new audio signal has a sampling rate common in today's multimedia communications (e.g., 48 kHz), server 102 downsamples the new audio signal to a lower sampling rate, e.g., 16 kHz, which is used to train the feature extraction model before running it. Server 102 then generates an improved signal from the feature vector set using a WaveRNN model. In this case, the samples used to generate the current sample of the improved audio signal are the previous samples generated for the improved audio signal. The generation process is autoregressive, by randomly sampling from a predicted distribution of current samples based on previously generated samples.

[0067] In some embodiments, as described above, the PASE+ model can be replaced with an alternative feature extraction model, or the WaveRNN model can be replaced with an alternative speech enhancement model, when generating an improved speech signal from a given new contaminated coded speech signal.

[0068] In some embodiments, the server 102 transmits the improved audio signal to an output device, such as an audio player or other computing device, which can further process the improved audio signal.

[0069] 5. Exemplary Procedure

[0070] Figure 4 illustrates an exemplary process performed by a voice management computer system according to some embodiments disclosed herein. Figure 4 is shown in a simplified schematic form for clarity, and other embodiments may include more, fewer, or different elements combined in various ways. Each of Figure 4 is intended to disclose an algorithm, plan, or outline that, when executed, can be used to implement one or more computer programs or other software elements that cause the functional improvements and technical advancements described herein. Furthermore, the flowcharts herein are described at the same level of detail that a person skilled in the art would normally use to communicate with one another about algorithms, plans, or specifications that form the basis of a software program that he plans to code or implement using his accumulated skills and knowledge.

[0071] In step 402, server 102 is programmed to retrieve coded audio data, including the first frameset.

[0072] In some embodiments, the server 102 is programmed to receive the original coded data and downsample it to obtain coded audio data. The original coded data may correspond to a sampling rate of 48 kHz, and the coded audio data may correspond to a sampling rate of 16 kHz. The coded audio data may include noise or reverb.

[0073] In step 404, server 102 is programmed to extract feature vector sets from coded audio data using a self-supervised deep learning model that includes a neural network, with each feature vector set being extracted from the first frame set.

[0074] In some embodiments, a self-supervised deep learning model includes an encoder and a plurality of workers. Each of the plurality of workers performs a self-supervised task related to an individual speech characteristic. One of the plurality of workers may perform a self-supervised task related to a predefined sampling strategy that extracts anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

[0075] In step 406, server 102 is programmed to generate enhanced speech data, including a second frameset, from a feature vector set using a generative deep learning model that includes a neural network, and the enhanced speech data corresponds to the clean speech in the coded audio data.

[0076] In some embodiments, the generative deep learning model includes a conditional network and a recurrent network. The conditional network transforms a feature vector set into an output feature vector set by considering multiple frames each time. The recurrent network generates enhanced speech data from the output feature vector set with one sample at a time, and each frame in a second frame set contains multiple samples. The recurrent network can generate new samples for each frame of the enhanced speech data using the corresponding feature vectors from the feature vector set and samples from the previously generated enhanced speech data.

[0077] In some embodiments, the server 102 is programmed to acquire a training set of distorted, downsampled audio signals to a predetermined sampling rate. The server is further configured to use the training set of distorted, downsampled audio signals to build a self-supervised deep learning model.

[0078] In some embodiments, the server 102 is programmed to acquire a second dataset of clean audio signals at a predetermined sampling rate corresponding to the coded and downsampled audio signals. The server is also configured to acquire a first training set by distorting a copy of the second dataset with one or more artifacts caused by the recording environment, recording equipment, or coding algorithm. The server is further configured to use the second dataset as well to build a self-supervised deep learning model.

[0079] In some embodiments, server 102 is programmed to acquire a dataset of coded, downsampled audio signals for a predetermined sampling rate. Server 102 is further configured to generate a training set of feature vectors from the dataset using a self-supervised deep learning model. Furthermore, server 102 is configured to build a generative deep learning model using the training set of feature vectors.

[0080] In some embodiments, the server 102 is programmed to acquire a third dataset of clean audio signals at a predetermined sampling rate corresponding to the coded and downsampled audio signals. The server is further configured to use the third dataset as well to build a generative deep learning model.

[0081] In some embodiments, the server 102 is programmed to acquire a dataset of distorted, downsampled audio signals to a predetermined sampling rate, and to use the dataset to train a coupled model that includes a self-supervised deep learning model connected to a generative deep learning model.

[0082] Various aspects of the disclosed embodiments may be apparent from the enumerated example embodiments (EEE) listed below.

[0083] A computer method for restoring clean audio from (EEE1) coded audio data, Steps include obtaining coded audio data including the first frameset, A step of extracting a set of feature vectors from the coded audio data using a self-supervised deep learning model including a neural network, wherein the set of feature vectors is extracted from each of the first frame sets, A step of generating enhanced speech data, including a second frameset, from the feature vector set using a generative deep learning model including a neural network, wherein the enhanced speech data corresponds to the clean speech in the coded audio data, A method that includes this.

[0084] (EEE2) A computer implementation of EEE1, further comprising the step of receiving original coded data, wherein the receiving step includes the step of downsampling the original coded data.

[0085] (EEE3) The computer implementation method according to EEE2, wherein the original coded data corresponds to a sampling rate of 48kHz and the coded audio data corresponds to a sampling rate of 16kHz.

[0086] (EEE4) The computer implementation method described in any of EEE1 to 3, wherein the coded audio data includes noise or reverb.

[0087] (EEE5) A computer implementation method according to any of EEE~4, wherein the self-supervised deep learning model comprises an encoder and a plurality of workers, each of the plurality of workers performing a self-supervised task relating to a distinct speech characteristic, and each of the plurality of workers performing a self-supervised task relating to a predefined sampling strategy, which extracts anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

[0088] (EEE6) The computer implementation method according to any one of EEE~5, wherein the generative deep learning model includes a conditional network and a recurrent network, the conditional network converts the feature vector set into an output feature vector set by considering multiple frames each time, the recurrent network generates the enhanced speech data from the output feature vector set one sample at a time, and each frame of the second frame set includes multiple samples.

[0089] (EEE7) A step of obtaining a training set of distorted audio signals at a specific sampling rate lower than a given sampling rate, The steps include constructing the self-supervised deep learning model using the aforementioned training set of distorted audio signals, A computer implementation method described in any of EEE-6, further including the above.

[0090] (EEE8) The steps include obtaining a dataset of audio signals downsampled to a predetermined sampling rate, The steps include generating a training set of feature vectors from the dataset using the self-supervised deep learning model, The steps include constructing the generative deep learning model using the training set of the feature vector set, A computer implementation method described in any of EEE-7, further including the above.

[0091] (EEE9) A step of obtaining a dataset of distorted audio signals at a specific sampling rate lower than a given sampling rate, A step of training a coupled model, which includes the self-supervised deep learning model connected to the generative deep learning model, using the dataset; A computer implementation method described in any of EEE-8, further including the above.

[0092] A system for restoring clean audio from (EEE10) coded audio data, Memory and One or more processors coupled to the memory and configured to perform a computer implementation method described in any of EEE to 9, A system that includes this.

[0093] (EEE11) A computer-readable non-temporary storage medium storing computer-executable instructions, wherein, when executed, the computer-executable instructions perform a method for restoring clean audio from coded audio data, and the method is A step of obtaining a dataset of coded audio signals that have been downsampled to a predetermined sampling rate, The steps include generating a training set of feature vectors from the dataset using the self-supervised deep learning model, The steps include constructing a generative deep learning model using the training set of the aforementioned feature vector set, The steps include: extracting a set of feature vectors from coded audio data using the self-supervised deep learning model described above; The steps include generating enhanced speech data from the feature vector set using the generative deep learning model, Computer-readable non-temporary storage media, including [specific type of storage medium].

[0094] (EEE12) The above method is The steps include obtaining a first training set of coded audio signals at a specific sampling rate lower than the predetermined sampling rate, The steps include generating the self-supervised deep learning model from the first training set, A computer-readable non-temporary storage medium as described in EEE1, further including the above.

[0095] (EEE13) The above method is The steps include obtaining a second dataset of clean audio signals at a specific sampling rate corresponding to the coded audio signal, The steps include obtaining the first training set by distorting a copy of the second dataset with one or more artifacts caused by the recording environment, recording device, or coding algorithm, It further includes, The generation step is performed using the second dataset, in the computer-readable non-temporary storage medium described in EEE2.

[0096] (EEE14) The method further includes the step of receiving the original coded data at the predetermined sampling rate, The extraction step includes a step of downsampling the original coded data, wherein the computer-readable non-temporary storage medium is as described in any of EEE1 to 13.

[0097] (EEE15) A computer-readable non-temporary storage medium according to any one of EEE1 to 14, wherein the dataset of the downsampled coded audio signal includes noise or reverb, and the coded audio data also includes noise or reverb.

[0098] (EEE16) A computer-readable non-temporary storage medium according to any one of EEE1 to 15, wherein the self-supervised deep learning model comprises an encoder and a plurality of workers, each of which worker performs a self-supervised task relating to a distinct speech characteristic, and the worker of which performs a self-supervised task relating to a predefined sampling strategy, which extracts anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

[0099] (EEE17) A computer-readable non-temporary storage medium according to any one of EEE1 to 16, wherein the coded audio data includes a first frame set, the feature vector sets are each extracted from the first frame set, and the enhanced audio data includes a second frame set.

[0100] (EEE18) The computer-readable non-temporary storage medium according to EEE7, wherein the generative deep learning model includes a conditional network and a recurrent network, the conditional network converts a feature vector set from the feature vector set into an output feature vector set by considering multiple frames each time, the recurrent network generates the enhanced speech data from the output feature vector set one sample at a time, and each frame of the second frame set includes multiple samples.

[0101] (EEE19) The computer-readable non-temporary storage medium according to EEE8, wherein the recurrent network generates a new sample for each frame of the enhanced speech data using the corresponding feature vectors of the feature vector set and previously generated samples of the enhanced speech data.

[0102] (EEE20) The above method is The step further includes obtaining a second dataset of clean audio signals at a predetermined sampling rate corresponding to the downsampled coded audio signal, The above construction step is further performed using the second dataset, for a computer-readable non-temporary storage medium as described in any of EEE1 to 19.

[0103] 6. Hardware Implementation

[0104] According to one embodiment, the technology described herein is implemented by at least one computing device. The technology may be implemented whole or in part using a combination of at least one server computer and / or other computing devices connected by a network such as a packet data network. The computing device may include at least one digital electronic device such as an application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) that is hard-wired or permanently programmed to perform the technology, or it may include firmware, memory, other storage devices, or at least one general-purpose hardware processor programmed to perform the technology according to program instructions in the combination. Such a computing device may combine hard-wired logic, ASICs, or FPGAs having custom programs for achieving the described technology. Computing devices may include server computers, workstations, personal computers, portable computer systems, handheld devices, mobile computing devices, wearable devices, body-worn or implantable devices, smartphones, smart home appliances, internetworking devices, autonomous or semi-autonomous devices such as robots or unmanned ground or air vehicles, any other electronic devices incorporating hard connections and / or program logic to implement the described technologies, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0105] Figure 5 is a block diagram illustrating an exemplary computer system in which an embodiment may be implemented. In the example of Figure 5, instructions for implementing the technology of the disclosure in the computer system 500 and hardware, software, or combination of hardware and software are shown schematically, for example, as boxes and circles, with respect to communications relating to the computer architecture and computer system implementation, at the same level of detail commonly used by those skilled in the art related to the disclosure.

[0106] The computer system 500 includes an input / output (I / O) subsystem 502 which may include buses and / or other communication mechanisms for communicating information and / or instructions between components of the computer system 500 via electronic signal paths. The I / O subsystem 502 may include an I / O control unit, a memory control unit, and at least one I / O port. The electronic signal paths are schematically shown in the figure, for example, as lines, one-way arrows, or two-way arrows.

[0107] At least one hardware processor 504 is connected to the I / O subsystem 502 for processing information and instructions. The hardware processor 504 may include, for example, a general-purpose microprocessor or microcontroller and / or an embedded system or graphics processing unit (GPU) or a dedicated microprocessor such as a digital signal processor or an ARM processor. The processor 504 may include an integrated arithmetic logic unit (ALU) or be coupled to a separate ALU.

[0108] The computer system 500 includes one or more units of memory 506, such as main memory, which is connected to the I / O subsystem 502 for electronically and digitally storing data and instructions to be executed by the processor 504. Memory 506 may include volatile memory such as various forms of random access memory (RAM) or other dynamic storage devices. Memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 504. When such instructions are stored in a non-temporary computer-readable recording medium accessible to the processor 504, the computer system 500 can be made a dedicated machine customized to perform the operations specified in the instructions.

[0109] The computer system 500 further includes non-volatile memory such as read-only memory (ROM) 508 or other static memory connected to the I / O subsystem 502 for storing information and instructions for the processor 504. The ROM 508 may include various forms of programmable ROM (PROM), such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). The permanent memory unit 510 may include various forms of non-volatile RAM (NVRAM), such as flash memory or solid-state memory, magnetic disks, or optical disks such as CD-ROMs or DVD-ROMs, and may be connected to the I / O subsystem 502 for storing information and instructions. The memory unit 510 is an example of a non-volatile computer-readable medium that may be used to store instructions and data that, when executed by the processor 504, cause the computer to perform the techniques of this specification.

[0110] Instructions in memory 506, ROM 508, or storage device 510 may include one or more instruction sets organized as modules, methods, objects, functions, routines, or calls. Instructions may be organized as one or more computer programs, operating system services, or application programs including mobile applications. Instructions may include operating systems and / or system software; one or more libraries supporting multimedia, programming, or other functions; data protocol instructions or stacks for implementing TCP / IP, HTTP, or other communication protocols; file processing instructions for interpreting and rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a graphical user interface (GUI), command-line interface, or text user interface; and application software such as office suites, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or various other applications. Instructions may implement web servers, web application servers, or web clients. Instructions may be organized as a presentation layer, an application layer, and a data storage layer such as a relational database system using a structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0111] The computer system 500 may be connected to at least one output device 512 via an I / O subsystem 502. In one embodiment, the output device 512 is a digital computer display. Examples of displays that may be used in various embodiments include a touchscreen display or a light-emitting diode (LED) display or a liquid crystal display (LCD) or an electronic paper display. The computer system 500 may include other types of output devices 512 as an alternative to or in addition to the display device. Examples of other types of output devices 512 include a printer, a ticket printer, a plotter, a projector, an audio card or video card, a speaker, a buzzer, or a piezoelectric element or other audible device, a lamp or an LED or LCD indicator, a haptic device, an actuator or a servo.

[0112] At least one input device 514 is connected to the I / O subsystem 502 to communicate signals, data, command selections, or gestures to the processor 504. Examples of input devices 514 include touchscreens, microphones, still or video digital cameras, alphanumeric and other keys, keypads, keyboards, graphic tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and / or various types of sensors such as force sensors, motion sensors, thermal sensors, accelerometers, gyroscopes, inertial measurement unit (IMU) sensors, and / or various types of radio transceivers such as cellular or Wi-Fi, radio frequency (RF) or infrared (IR) transceivers and Global Positioning System (GPS) transceivers.

[0113] Another type of input device is a control device 516 that can perform cursor control or other child control functions, such as navigation within a graphical interface on a display screen, as an alternative to or in addition to the input function. The control device 516 may be a touchpad, mouse, trackball, or cursor directional keys for communicating directional information and command selection to the processor 504 and for controlling cursor movement on the display 512. The input device may have at least two degrees of freedom in two axes, i.e., a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane. Another type of input device is a wired, wireless, or optical control device such as a joystick, wand, console, steering wheel, pedal, gear shift mechanism, or other type of control device. The input device 514 may include a combination of several different input devices, such as a video camera and a depth sensor.

[0114] In another embodiment, the computer system 500 may include an Internet of Things (IoT) device, in which one or more of the output device 512, input device 514, and control device 516 are omitted. Alternatively, in such an embodiment, the input device 514 may include one or more cameras, motion detectors, thermometers, microphones, seismic detectors, other sensors or detectors, measuring devices or encoders, and the output device 512 may include a dedicated display such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator or a servo.

[0115] When the computer system 500 is a mobile computing device, the input device 514 may include a Global Positioning System (GPS) receiver connected to a GPS module capable of triangulating multiple GPS satellites and determining and generating positional data such as latitude and longitude values ​​of a geographical location or the geophysical location of the computer system 500. The output device 512 may include hardware, software, firmware, and interfaces for generating position reporting packets, notifications, pulses or heartbeat signals, or other circular data transmissions that specify the location of the computer system 500, either alone or in combination with other purpose-specific data, directed to the host 524 or server 530.

[0116] The computer system 500 may implement the technology described in this specification using customized hard-connected logic, at least one ASIC or FPGA, firmware and / or program instructions or logic that, when loaded and used or executed in combination with the computer system, cause or program the computer system to operate as a dedicated machine. According to one embodiment, the technology described in this specification may be executed by the computer system 500 in response to the processor 504 executing at least one sequence of at least one instruction contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as a storage device 510. The execution of the instruction sequence contained in the main memory 506 causes the processor 504 to perform the processing steps described in this specification. In an alternative embodiment, hard-connected circuits may be used instead of or in combination with software instructions.

[0117] As used in this specification, the term “storage medium” refers to any non-temporary medium that stores data and / or instructions causing a machine to operate in a particular manner. Such storage mediums may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks such as storage device 510. Volatile media include dynamic memory such as memory 506. General forms of storage mediums may include, for example, hard disks, solid drives, flash drives, magnetic data storage media, any optical or physical data storage media, memory chips, etc.

[0118] A storage medium is distinct from a transmission medium, but may be used in conjunction with it. A transmission medium is involved in transferring information between storage mediums. For example, a transmission medium includes coaxial cables, copper wires, and optical fibers, and includes wires, including the bus of the I / O subsystem 502. A transmission medium can also take the form of acoustic or optical waves, such as those generated between radio waves and infrared data communications.

[0119] Various forms of media may be involved in transmitting at least one sequence of at least one instruction to the processor 504 for execution. For example, the instruction may first be transmitted on a magnetic disk or solid drive of a remote computer. The remote computer may load the instruction into its dynamic memory and transmit the instruction using a modem over a communication link such as an optical fiber, coaxial cable, or telephone line. A modem or router located locally of the computer system 500 may receive data over the communication link and convert the data so that it can be read by the computer system 500. For example, a receiver such as a radio frequency antenna or infrared detector may receive data transmitted in a radio or optical signal, and appropriate circuitry may provide the data to the I / O subsystem 502, for example, by placing the data on a bus. The I / O subsystem 502 transmits the data to memory 506 and reads and executes the processor 504h instruction from memory 506. The instruction received by memory 506 may optionally be stored in storage device 510 before or after execution by the processor 504.

[0120] The computer system 500 also includes a communication interface 518 connected to bus 502. The communication interface 518 provides two-way data communication coupled to a network link 520 that is directly or indirectly connected to at least one communication network, such as network 522 or a public or private cloud on the Internet. For example, the communication interface 518 may be an Ethernet network interface, an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connectivity to a corresponding type of communication line, such as an Ethernet cable or any type of metal cable or fiber optic line or telephone line. Network 522 broadly represents a local area network (LAN), a wide area network (WAN), a campus network, an internetwork, or any combination thereof. The communication interface 518 may include a LAN card to provide data communication connectivity to a compatible LAN, or a cellular radiophone interface wired to transmit or receive cellular data according to cellular radiophone radio networking standards, or a satellite radio interface wired to transmit or receive digital data according to satellite radio networking standards. In any such implementation, the communication interface 518 transmits and receives electrical, electromagnetic, or optical signals via a signal path that carries digital data streams representing various types of information.

[0121] Network link 520 typically provides electrical, electromagnetic, or optical data communication to other data devices directly or through at least one network, for example, using satellite, cellular, Wi-Fi, or Bluetooth® technology. For example, network link 520 may provide connectivity to host computer 524 through network 522.

[0122] Furthermore, network link 520 may provide connectivity via network 522 or via internetworking equipment and / or computers to other computer devices operated by an Internet Service Provider (ISP) 526. ISP 526 provides data communication services through a worldwide packet data communication network represented as the Internet 528. Server computer 530 may be connected to the Internet 528. Server 530 broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or computer running a containerized program system such as Docker or Kubernetes. Server 530 may represent an electronic digital service implemented using multiple computers or instances, accessed and used by sending web service requests, URL (Uniform Resource Locator) strings containing HTTP payload parameters, application programming interface (API) calls, app service calls, or other service calls. Computer system 500 and server 530 may form elements of a distributed computing system, including other computers, processing clusters, server farms, or other organizations of computers cooperating to perform tasks or run applications or services. Server 530 may include one or more instruction sets organized as modules, methods, objects, functions, routines, or calls. Instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps.The instructions may include operating system and / or system software; one or more libraries supporting multimedia, programming, or other functions; data protocol instructions or stacks for implementing TCP / IP, HTTP, or other communication protocols; file format processing instructions for interpreting or rendering files coded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for a graphical user interface (GUI), command-line interface, or text user interface; and application software such as office suites, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or various other applications. Server 530 may include a presentation layer, an application layer, and a web application server hosting a data storage layer such as a relational database system using a structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0123] The computer system 500 can send messages, including program code, and receive data and instructions via a network, network link 520, and communication interface 518. In the example of the internet, server 530 may send the code necessary for an application program via the internet 528, ISP 526, local network 522, and communication interface 518. The received code, once received and / or stored in storage device 510 or other non-volatile storage device for later execution, may be executed by processor 504.

[0124] The execution of instructions described in this chapter can be implemented as a process in the form of an instance of a running computer program, consisting of program code and its current activity. Depending on the operating system (OS), a process may consist of multiple execution threads that execute instructions concurrently. In this context, a computer program is a passive collection of instructions, while a process may be the actual execution of those instructions. Multiple processes may be associated with the same program; for example, opening multiple instances of the same program often means that multiple processes are running. Multitasking can be implemented to allow multiple processes to share a processor 504. While each processor 504 or processor core executes a single task at a time, the computer system 500 may be programmed to implement multitasking so that each processor can switch between running tasks without waiting for each task to finish. In embodiments, switching may occur when a task performs an input / output operation, when a task indicates that it is switchable, or by a hardware interrupt. Time-sharing can be implemented to speed up the response of interactive user applications by rapidly performing context switching and making it appear as if multiple processes are running concurrently. In this embodiment, for security and reliability, the operating system prevents direct communication between independent processes and provides strictly mediated and controlled inter-process communication functionality.

[0125] 7. Expansion and Replacement

[0126] In the above specification, embodiments of the disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings should therefore be considered illustrative, not restrictive. The sole and exclusive indication of the scope of the disclosure, and what the applicant intends to be the scope of the disclosure, is the literal equivalent of the claims issued in any specific form from this application, including any subsequent amendments.

Claims

1. A computer method for restoring clean audio from coded audio data, A step of receiving a feature vector set from the coded audio data, wherein the feature vector set is extracted from a first frame set using a deep learning model including a neural network. A step of generating enhanced speech data, including a second frameset, from the feature vector set using a generative deep learning model including a neural network, wherein the enhanced speech data corresponds to the clean speech in the coded audio data. Includes, The generative deep learning model includes a conditional network and a recurrent network, and is implemented by a computer.

2. The computer implementation method according to claim 1, wherein the deep learning model includes an encoder.

3. The computer implementation method according to claim 1, wherein the generative deep learning model is configured to operate at a sampling rate of 16 kHz.

4. The deep learning model further includes multiple workers, Each of the aforementioned workers is configured to perform a self-supervised task related to a separate speech characteristic. The computer implementation method according to claim 2, wherein one of the plurality of workers is configured to perform a self-supervised task related to a predefined sampling strategy, which extracts anchor samples, positive samples, and negative samples from a pool of representations generated by the encoder.

5. The first step is to receive the original coded data, The steps include downsampling the original coded data, The computer implementation method according to claim 1, further comprising:

6. The computer implementation method according to claim 5, wherein the original coded data corresponds to a sampling rate of 48 kHz, and the coded audio data corresponds to a sampling rate of 16 kHz.

7. The computer implementation method according to claim 1, wherein the coded audio data includes noise or reverb.

8. The computer implementation method according to claim 1, wherein the conditional network converts the feature vector set into an output feature vector set by considering multiple frames each time, the recurrent network generates the enhanced speech data one sample at a time from the output feature vector set, and each frame of the second frame set includes multiple samples.

9. A step of obtaining a training set of distorted audio signals at a specific sampling rate lower than a given sampling rate, The steps include constructing the deep learning model using the training set of distorted audio signals, The computer implementation method according to claim 1, further comprising:

10. The steps include obtaining a dataset of audio signals downsampled to a predetermined sampling rate, The steps include generating a training set of feature vectors from the dataset using the deep learning model described above, The steps include constructing the generative deep learning model using the training set of the feature vector set, The computer implementation method according to claim 1, further comprising:

11. The steps include obtaining a dataset of distorted audio signals at a specific sampling rate lower than a predetermined sampling rate, A step of training a coupled model that includes the deep learning model connected to the generative deep learning model using the dataset, The computer implementation method according to claim 1, further comprising:

12. A system for restoring clean audio from coded audio data, Memory and One or more processors coupled to the memory and configured to perform the computer implementation method according to any one of claims 1 to 11, A system that includes this.

Images