BANDWIDTH EXTENSION AND SPECTRAL BALANCE IN THE FEATURE DOMAIN FOR ASR DATA EXTENSION

Description

BACKGROUND OF THE DISCLOSURE 1. Field of the Disclosure

[0001] The present disclosure relates to a system and a method for speech processing, and relates more particularly to a system and a method for feature domain artificial bandwidth extension (ABE) in speech processing.2. Description of the Related Art

[0002] For training or adapting a speech processing system (SPS), e.g., automatic speech recognition (ASR) or voice biometrics system, a situation arises in which a first set of data collected from a certain processing bandwidth A is sought to be used for training or adapting a speech processing system that is expected to process data using a higher bandwidth B. A processing bandwidth refers to the signal processing that is applied to the acoustic signal captured, e.g., at a microphone. An example of signal processing is band-pass filtering typically applied before transmission or storage, which band-pass filtering is applied to limit the frequency range in which information is present in a signal (e.g., for compression reasons). As an example, it is typical to refer to an 8kHz sample rate as narrowband and 16kHz sample rate as wideband, e.g., in the telephony use case. In a typical example, data from a first signal source (e.g., corresponding to processing bandwidth A) may be required for performing data augmentation to further match its acoustic properties to a second signal source (e.g., corresponding to processing bandwidth B), for example by applying a room impulse response. A related issue that needs to be addressed is that of spectral balance (i.e., distribution of the long term energy in frequency spectrum) mismatch between data from signal source having processing bandwidth A and data from signal source having processing bandwidth B.

[0003] Current state of the art (SOTA) methods for dealing with the bandwidth mismatch (e.g., using lower-bandwidth-processed data to train and / or adapt higher-bandwidth-processed data) are referred to as artificial bandwidth extension (ABE) methods (algorithms), of which there are several known ones. The current SOTA ABE methods are focused on the telephony use case in which a narrowband signal is to be extended to a wideband signal. In such a case, the optimization criterion is human perception. More generally, all current ABE methods target a perceptual optimization criterion. However, for the speech processing (e.g., ASR) use case, this perceptual criterion may not be optimal. In addition, in the current SOTA, the spectral balance mismatch issue is typically addressed by equalizing the spectral balance to the desired shape, which may not be optimal for the speech processing (e.g., ASR) use case. Moreover, in the current SOTA, the processing systems are operating in a run time mode, meaning that they are typically required to operate with low latency and low computational complexity and do not assume knowledge of the speech content (e.g., text transcription) in the signal.

[0004] Therefore, there is a need to provide an improved ABE which is optimal for the speech processing (e.g., ASR) use case. GAO JIANQING ET AL, "Mixed-Bandwidth Cross-Channel Speech Recognition via Joint Optimization of DNN-Based Bandwidth Expansion and Acoustic Modeling", 13 December 2018 describes that automatic speech recognition (ASR) systems are often built using scene related speech data due to large variations of transmission channels and sampling rates in different scenarios. In this study, they propose a general framework that establishes a unified model for diversified speech data with different sampling rates and channels. The framework is a joint optimization of deep neural network (DNN)-based bandwidth expansion and acoustic modeling to exploit a large amount of diversified training data. First, we design two novel DNN architectures to map the acoustic features from narrowband to wideband speech through direct mapping and progressive mapping. The learning targets of the direct mapping DNN (DNN-DM) are the acoustic features extracted from speech with the largest bandwidth, while the acoustic features from speech with all the other bandwidths are used as input. A progressive stacking network (PSN) gradually maps the features from the low sampling rates to the highest sampling rate through the design of intermediate target layers via multitask training. Then, in addition to these bandwidth expansion networks, they investigate several joint training strategies for DNN-based acoustic models. The experiments conducted on three diversified large-scale Mandarin speech datasets with different recording channels and sampling rates (6, 8, and 16 kHz) show that the proposed unified model using PSN for bandwidth expansion not only is a more flexible and compact design than conventional multiple acoustic models with each bandwidth for a specific sampling rate, but also yields consistent and significant improvements over bandwidth-dependent models with an average relative word error rate reduction of 6.2%, indicating that the proposed model can fully utilize the diversified cross-channel speech data with multiple bandwidths. Moreover, the proposed methods are verified to be robust on different realistic scenes and can be effectively extended to a long short-term memory framework. JIANQING GAO ET AL, "An experimental study on joint modeling of mixed-bandwidth data via deep neural networks for robust speech recognition", 2016 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), IEEE,24 July 2016 describes joint modeling strategies leveraging upon large-scale mixed-band training speech for recognition of both narrowband and wideband data based on deep neural networks (DNNs). They utilize conventional down-sampling and up-sampling schemes to go between narrowband and wideband data. They also explore DNN-based speech bandwidth expansion (BWE) to map some acoustic features from narrowband to wideband speech. By arranging narrowband and wideband features at the input or the output level of BWE-DNN, and combining down-sampling and up-sampling data, different DNNs can be established. The experiments on a Mandarin speech recognition task show that the hybrid DNNs for joint modeling of mixed-band speech yield significant performance gains over both the narrowband and wideband speech models, well-trained separately, with a relative character error rate reduction of 7.9% and 3.9% on narrowband and wideband data, respectively. Furthermore, the proposed strategies also consistently outperform other conventional DNN-based methods. US20080215322A1 describes a method and a system for generating training data (D T ) for an automatic speech recogniser (2) for operating at a particular first sampling frequency (f H ), comprising steps of deriving spectral characteristics (S L ) from audio data (D L ) sampled at a second frequency (f L ) lower than the first sampling frequency (f H ), extending the bandwidth of the spectral characteristics (S L ) by retrieving bandwidth extending information OBE) from a codebook (6), and processing the bandwidth extended spectral characteristics (S LE ) to give the required training data (D T ). Moreover a method and a system (5) for generating a codebook (6) for extending the bandwidth of spectral characteristics (S L ) for audio data (D L ) sampled at a second sampling frequency (f L ) to spectral characteristics (S H ) for a first sampling frequency (f H ) higher than the second sampling frequency (f L ) are described. NIDADAVOLU PHANI SANKAR ET AL, "Investigation on Neural Bandwidth Extension of Telephone Speech for Improved Speaker Recognition", ICASSP 2019 - 2019 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), IEEE, 12 May 2019 describes previous work on training mixed-bandwidth (BW) speaker recognition system by predicting missing information in upperband (UB) of upsampled telephone speech. Mixed-BW systems combine speech from narrowband (NB) and wideband (WB) speech corpora by basic upsampling of NB speech with low-pass filter interpolator, resulting in no information loss in the original WB speech. In this work, they explore the usage of a deep residual full-convolutional neural network (CNN) and a bidirectional long short term memory (BLSTM) network along with a previously proposed deep neural network (DNN) for bandwidth extension (BWE) of NB telephone speech. Speaker recognition systems trained with bandwidth extended features improved in performance over mixed-BW and NB baseline systems. In terms of detection cost function (DCF), the CNN-BWE system improved by 10.78% and 15.96% (relative) in the Speakers In The Wild (SITW) eval core and assist-multi-speaker condition respectively w.r.t. the NB baseline; and improved by 3.21% and 4.13% w.r.t. to the mixed-BW baseline. PETER BELL ET AL, "Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview", ARXIV. ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853,28 February 2021 a structured overview of adaptation algorithms for neural network-based speech recognition, considering both hybrid hidden Markov model / neural network systems and end-to-end neural network systems, with a focus on speaker adaptation, domain adaptation, and accent adaptation. The overview characterizes adaptation algorithms as based on embeddings, model parameter adaptation, or data augmentation. A meta-analysis of the performance of speech recognition adaptation algorithms is presented, based on relative error rate reductions as reported in the literature. IGOR SZOKE ET AL, "Building and Evaluation of a Real Room Impulse Response Dataset", ARXIV. ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853,16 November 2018 describes BUT ReverbDB - a dataset of real room impulse responses (RIR), background noises and re-transmitted speech data. The retransmitted data includes LibriSpeech test-clean, 2000 HUB5 English evaluation and part of 2010 NIST Speaker Recognition Evaluation datasets. A detailed description of RIR collection (hardware, software, post-processing) is provided that can serve as a "cook-book" for similar efforts. They also validate BUT ReverbDB in two sets of automatic speech recognition (ASR) experiments and draw conclusions for augmenting ASR training data with real and artificially generated RIRs. It is shown that a limited number of real RIRs, carefully selected to match the target environment, provide results comparable to a large number of artificially generated RIRs, and that both sets can be combined to achieve the best ASR results. The dataset is distributed for free under a non-restrictive license and it currently contains data from 8 rooms, which is growing. The distribution package also contains a Kaldi-based recipe for augmenting publicly available AMI close-talk meeting data and test the results on an AMI single distant microphone set, allowing it to reproduce our experiments.SUMMARY OF THE DISCLOSURE

[0005] The invention is set out in the appended set of claims.

[0006] According to an example embodiment of a method and a system for ABE, the transcription information typically available with ASR training / adaptation data is utilized to achieve an improved ABE processing, e.g., with an artificial intelligence (AI) model that has an optional ASR-related cost function.

[0007] According to an example embodiment of a method and a system for ABE, the AI model is used to apply the ABE processing to low-bandwidth data, and then optionally, augment the ASR training / adaptation data with a range of bandwidths, thereby enabling the ASR model to learn to work with different bandwidth signals.

[0008] According to an example embodiment of a method and a system for ABE, a range of equalizations (i.e., a number of equalization filtering operations) can be applied to the long term spectrum and enable the ASR AI model to learn to selectively ignore or be more robust to different spectral balance issues, thereby enabling the ASR AI model to effectively operate in a range of acoustic environments. In this manner, a more generic ASR AI model can be effectively deployed in a range of scenarios involving acoustic changes in the deployed environment. According to an example embodiment of the method and the system for ABE, the ABE process is implemented with ASR optimization criteria (e.g., minimum word error rate (WER)) and operate in the feature domain used for ASR (rather than in the waveform domain or linear frequency domain). In this manner, the ABE process optimization is attuned to ASR and with potentially less complexity (e.g., when performing the ABE in the log-Mel filter bank domain, the dimension of the problem can be as low as 60 dimensions compared to 256 dimensions for linear frequency domain for each time frame).

[0009] According to an example embodiment of the method and the system for ABE, in the use case for augmentation of training / adaptation data, the time-aligned and transcribed text corresponding to the audio is utilized to take advantage of text-to-speech (TTS) technology to assist in the reconstruction of the higher frequency bandwidth regions (i.e., the reconstruction is based on an acoustic component as well as a linguistic component).

[0010] According to an example embodiment of the method and the system for ABE, the training / adaption processing is performed offline, e.g., utilizing large scale computing resources, such that there is no requirement for low complexity or delay in processing.

[0011] According to an example embodiment of the method and the system for ABE, an AI model is trained to learn to reconstruct the high frequency bandwidth signal components from a lower frequency bandwidth signal using text, acoustic data and / or ASR optimization criteria. The reconstructed high frequency bandwidth data can be low-pass filtered with different cut off frequencies to augment the data and allow the AI model to work in a variety of signal bandwidth environments.BRIEF DESCRIPTION OF THE FIGURES

[0012] FIG. 1 illustrates two different processing bandwidths for audio signal data. FIG. 2a illustrates an example method of training a TTS-based ABE model. FIG. 2b illustrates the details of an example embodiment of a speaker embedding system not encompassed by the wording of the claims. FIG. 2c illustrates generation of training data from lower bandwidth domain A data. FIG. 2d illustrates HFRN-processed domain A data and the reconstructed domain B data being used to train a new ASR system. FIG. 3a illustrates the characteristic shape of LTASS. FIG. 3b shows an example embodiment in which the LTASS-based equalization transfer function is applied. FIG. 3c shows an example process for obtaining the PLTLDs for domain A and domain B data sets. FIG. 4 illustrates the equalization transfer function being multiplied with a random perturbation vector to produce a new equalization transfer function. DETAILED DESCRIPTION

[0013] FIG. 1 illustrates two different processing bandwidths (0-4 kHz, labeled "4 kHz bandwidth", and 0-8 kHz, labeled "8 kHz bandwidth") for audio signal data. For training or adapting a speech processing system, e.g., automatic speech recognition (ASR) or voice biometrics system, a first example scenario involves a first set of data collected from a lower processing bandwidth A (e.g., 0-4 kHz processing bandwidth), also referred to as data domain A, which first set of data is sought to be used for training or adapting a speech processing system that is expected to process data using a higher bandwidth (e.g., within the 0-8 kHz bandwidth), also referred as data domain B. In this example scenario, data domain A is potentially cleaner, i.e., has less reverberation and noise, and has a different spectral tilt (i.e., distribution of long-term energy in frequency domain) than data domain B, but the data domain B is the domain in which the ASR system is expected to operate.

[0014] A first example embodiment not encompassed by the wording of the claims provides a method and a system for using lower-bandwidth-processed data to train and / or adapt higher-bandwidth-processed data (which is referred to as artificial bandwidth extension (ABE)) focuses on creating an ASR model (e.g., ASR AI model) that works well in data domain B using data from data domain A. To achieve this goal, in a first example embodiment of the method and the system for ABE, e.g., a text-to-speech (TTS) type ABE system, the ABE system is trained to learn to map the data from data domain A to data domain B in the feature space (e.g., the log-Mel filter bank (LMFB) spectrum or space) of the ASR model, along with a loss function that includes ASR loss and / or with additional speaker information. In addition, an example embodiment of the method and the system for ABE is trained to learn one or more equalization filter(s) for mapping the spectral tilt of data domain A to the spectral tilt of data domain B. Optionally, additional data augmentation techniques, e.g., room impulse response (RIR), can be applied.

[0015] In accordance with the first example embodiment of the method according to the present disclosure, an AI model (e.g., a neural network) is trained to learn to re-construct a given information of lower bandwidth data (of data domain A) in higher bandwidth domain (data domain B). An example method of training the AI model can include, without limitation, one or more of the following: 1) using a loss function that includes ASR loss and / or reconstruction loss; 2) using a text-to-speech component for reconstruction (i.e., linguistic information); 3) using speaker-related features for reconstruction, i.e., speaker-related meta tags such as gender, age, accent, language, or neural embeddings; 4) using the AI model to map lower bandwidth data to higher bandwidth data to match the respective spectral tilts; and 5) using other acoustic information, e.g., the location of the recording (room type, location within a room, etc.). In the first example embodiment of the method, which is explained in further detail below, a TTS-based ABE system is trained to learn to map the data from data domain A to data domain B in the feature space (e.g., the log-Mel filter bank space) of the ASR system, with i) a loss function that includes ASR loss, and ii) additional speaker information. Additionally, the TTS-based ABE system is trained to learn an equalization filter for mapping the spectral tilt from domain A to B.

[0016] FIG. 2a illustrates an example method of training a TTS-based ABE model, e.g., a TTS-based High Frequency Reconstruction Network (HFRN) 1001, to learn to re-construct the feature information in the higher frequencies given lower bandwidth input. The initial step involves processing (e.g., down sampling) the domain B data to match the lower bandwidth of data domain A. The domain B (i.e., high bandwidth) data are denoted as wb; the down-sampled version of the domain B data are denoted as nb; the ASR features from the true (unmodified) domain B data are denoted as Xwb; and the ASR features from the down-sampled version of the domain B data are denoted as Xnb (e.g., 80-dimensional log-Mel filter bank features). For example, in the case the domain B data are sampled at 16 kHz and the domain A data are sampled at 8 kHz, the domain B data (wb) are down-sampled to 8 kHz (e.g., by low-pass filtering followed by decimation) to produce nb. Then, from each of down-sampled data set B (nb) and data set A, both of which are now at a sample rate of 8kHz, a selected number of log-Mel filter bank features, e.g., 80, can be extracted. Essentially, what the method is attempting to create is a parallel data set from domain B data that includes the true data set B and an artificially bandwidth-reduced (via down-sampling) version of dataset B that has the lower bandwidth. This resulting parallel data set then allows us to train a neural network system (e.g., ASR AI system or model, which can be embodied at least in part as high frequency reconstruction network (HFRN)) to learn how to re-create the high frequency information. It should be noted that channel-related effects (such as those effecting the long term spectral shape) present in data set A may not be handled, so an additional (and optional) equalization step can be implemented to address these channel-related effects.

[0017] In addition to the above, Tnb denotes the text transcription (e.g., time-aligned phonemes and / or words) of down-sampled domain B data, and Mnb denotes one or more meta data embeddings including speaker and / or localization information, e.g., gender of the speaker, a d-Vector (denoting "deep vector") type neural embedding from a speaker recognition / verification system, and other meta data such as the location of the sound source, room type, Long Term Average Speech Spectrum (LTASS)-based equalization transfer functions, etc. The meta data embedding is described in further detail in connection with FIG. 2b.

[0018] FIG. 2b illustrates the details of an example embodiment of a speaker embedding system for extracting and learning speaker embedding vectors. The speaker embedding system 2001 shown in FIG. 2b includes a feature extraction module 2001a and a deep neural network 2001b system. The speaker embedding system 2001 extracts, e.g., the above-mentioned d-vector type embedding vectors for a given speech segment. The embedding vectors are vectors that describe the speaker characteristics such that voices that sound similar are close in the embedding space and those that are very different are far apart in the embedding space, whereby the embedding vectors allow a machine-learning system to distinguish among a large pool of voices. These embeddings are compressed representations of the speaker characteristics, which enable the HFRN to better reconstruct the missing high frequency data.

[0019] In addition to the above-described speaker embeddings, embeddings for other meta data associated with the speech data, e.g., the location of the sound source, room type, and LTASS-based equalization transfer function(s), can be provided. Alternatively, instead of embeddings for other meta data, a codebook-type vector for the other meta data can be provided. For example, the azimuth of sound sources can be discretized into a 5-bit binary vector that enables mapping the azimuth with a resolution of 360 / 2 5< (i.e., 11.25) degrees.

[0020] As shown in FIG. 2a, the TTS-based HFRN 1001 is trained to take selected input information (e.g., the acoustic information (Xnb), the text transcript information (Tnb) and the meta information (Mnb)) and produces an estimate of the higher bandwidth features (referred to as "reconstructed domain B data" in FIG. 2a), which is denoted as Ywb. In order to train the reconstruction network, HFRN 1001, an example embodiment can also take into account two loss terms: i) reconstruction loss, which measures the closeness of the reconstructed features (Ywb) to the actual high bandwidth features (Xwb); and ii) ASR loss, which measures the closeness of the ASR outputs from the reconstructed and actual high bandwidth features. The sum of the reconstruction loss (RL) and the ASR loss (AL) represents the total loss, which can be represented as the following expression: Total Loss = θ . RL + 1 − θ . AL where RL = L{Ywb,Xwb} and AL = L{W1,W2}. RL is the reconstruction loss in the feature space of ASR (e.g., log-Mel filter bank domain) and measures the closeness of the reconstructed features to the actual high bandwidth features. AL is the ASR loss that measures the closeness of the ASR outputs from the reconstructed and the actual high bandwidth features. L{} can be a suitable cost function, e.g., Root Mean Square Error (RMSE), Mean Square Error (MSE) or Mean Absolute Error (MAE). θ allows for controlling "bias" towards ASR target.

[0021] The WB ASR system 1002 shown in the right half of FIG. 2a is an ASR system pre-trained with the original (true) domain B data wb, which input of wb produces output W1, as shown in the upper right half of FIG. 2a. The pre-trained WB ASR system 1002 takes the reconstructed domain B data, Ywb, and produces output W2. The ASR loss for training the HFRN 1001 refers to the distance between the outputs W1 and W2. The reconstruction loss for training the HFRN 1001 is the closeness of Ywb to Xwb.

[0022] After the HFRN 1001 has been trained as discussed above, the next stage of the example method involves generating training data from domain A data (denoted as Xna), as shown in FIG. 2c. The following inputs are fed to the trained HFRN 1001: domain A acoustic features or information (Xna); Tna, which denotes the text transcription (e.g., time-aligned phonemes and / or words) of domain A data; and Mna, which denotes one or more meta data embeddings including Long Term Average Speech Spectrum (LTASS)-based equalization transfer functions and optionally speaker and / or localization information (such as gender, audio environment (e.g., the location of the sound source or room type). The trained HFRN 1001 outputs estimates of the high bandwidth features, Ywa, as HFRN-processed domain A data. Next, as shown in FIG. 2d, HFRN-processed domain A data, Ywa, and the reconstructed domain B data, Ywb, are used to train a new ASR system producing output W3, which ASR system is denoted as HFRN+WB ASR 2002.

[0023] In addition to the above, a process of applying Long Term Average Speech Spectrum (LTASS)-based equalization transfer function is implemented. The LTASS-based equalization transfer function describes how the spectral tilt in domain A maps to domain B. The LTASS has a characteristic shape, which is shown in FIG. 3a, that is used as a model for the clean speech spectrum, and hence the LTASS is often used in speech processing algorithms. The ITU-T P.50 standard (ITU-T, Artificial Voices, International Telecommunications Union (ITU-T) Recommendation P.50, September 1999) defines the equation for approximating the LTASS shown in Fig. 3a. The Power spectrum of Long term Deviation (PLD) for time frame i and frequency bin k is defined as: PLD i , k = log P s i , k − log P LTASS k , where P LTASS (k) is the LTASS power spectrum and P 8 (i, k) is the magnitude power spectrum of speech signal. P s (i, k) is defined as follows: P s i , k = S i , k × S * i , k where S(i, k) is the Discrete Fourier Transform (DTF) of the speech signal and S*(i, k) denotes the complex conjugate of the DFT of the speech signal. The speech signal is split into discrete short time frames by multiplying the time domain signal with a window function (e.g., Hanning window), producing I time frames (e.g., 30 ms frames). The long-term deviation of the magnitude spectrum of the signal (calculated over the entire dataset), P LTLD , is defined as follows P LTLD k = 1 N i ∑ i = 1 N i PLD i k where k is the frequency index, PLD is the power spectrum of long-term deviation as defined in (1).

[0024] FIG. 3b shows an example embodiment in which the LTASS-based equalization transfer function is applied. In this example embodiment, the difference between LTASS and the speech signal in domain A data set is computed, e.g., by averaging the spectra of speech from domain A data set and computing the distance from a typical LTASS curve. In other words, the difference between the signal of interest and LTASS in the log magnitude spectral domain is computed and average over the entire domain A data set. This results in the deviation of the long term spectrum in the domain A data set from LTASS (i.e., PLTLD discussed above). The difference between LTASS and the speech signal in domain B data set is computed in a similar manner. This process for obtaining the PLTLDs is illustrated in detail in FIG. 3c. Next, the two differences (i.e., PLTLDs) are divided to obtain a transfer function in the frequency domain (in the log-Mel filter bank ASR feature space, this would be a difference) that maps the spectral tilt from domain A to domain B. The resulting transfer function provides an additional meta data vector term that describes the equalization that needs to be applied to the input to ensure that the spectral tilt matches the spectral tilt in domain B. Each transfer function can be, e.g., a vector of 128 x 1 in dimension (if 32ms frames at 8kHz sample rate are assumed, 0.032*8000=256 samples would be provided, which, after applying a real Discrete Fourier Transform (DFT) or Fast Fourier Transform (FFT), result in a frequency spectrum with 128 complex coefficients, from which we can further obtain the power spectrum. Thus, the equalization system 3001 shown in FIG. 3b takes as inputs LTASS-based equalization transfer function (which is more precisely PLTLD-based transfer function, so this term is used interchangeably with LTASS-based equalization transfer function) and the feature inputs (i.e., reconstructed domain B data Ywb), and outputs the equalized features, reconstructed and equalized domain B data Ywb' (e.g., log-Mel filter bank features). It should be noted that this application of LTASS-based equalization transfer function is independent of the actual phoneme / word and is applied in a separate processing step.

[0025] In a second example scenario for training or adapting a speech processing system, e.g., automatic speech recognition (ASR) or voice biometrics system, a first set of data collected from 0-4 kHz processing bandwidth (also referred to as data domain A) and a second set of data collected from a higher bandwidth, e.g., within the 0-8 kHz bandwidth (also referred as data domain B), are present, and the goal of a second example embodiment of a method and a system for ABE is to create an ASR model that works well in both data domain A and data domain B. To achieve this goal, in the second example embodiment of the method and the system for ABE, e.g., a text-to-speech (TTS) type ABE system, the ABE system is trained in a manner substantially identical to the manner described in connection with the first example embodiment of the method and the system for ABE, with the addition of multi-condition training (MCT).

[0026] The above-described details of the first example embodiment apply to the second example embodiment, with the addition of multi-condition training (MCT) when training the ASR system. An example embodiment of the MCT involves two steps. As a first step, when computing the equalization transfer function from data domain A to data domain B, the equalization transfer function (e.g., PLTLD-based equalization transfer function, which is also referred to as LTASS-based equalization transfer function) is randomly perturbed (using random perturbation vectors) so that there are several equalizations learned (i.e., to account for a range of different spectral tilts). As shown in FIG. 4, according to the invention, the equalization transfer function is multiplied with a random perturbation vector to produce a new equalization transfer function, and this is repeated for different random perturbation vectors. As a second step, a number of HFRNs are trained with different cut-off frequencies to allow the ASR system to learn multiple different bandwidths. For example, 4 different HFRNs can be trained: i) one where the domain B data (e.g., 16kHz bandwidth) is down-sampled to 10 kHz and a corresponding HFRN trained; ii) another one where the domain B data is down-sampled to 8 kHz and a corresponding HFRN trained; iii) another one where the domain B data is down-sampled to 7 kHz and a corresponding HFRN trained; and iv) another one where the domain B data is down-sampled to 6 kHz and a corresponding HFRN trained. Then, the data from domain A are processed with a mix of the above 4 HFRNs to produce HFRN-processed domain A data (e.g., as shown in FIG. 2c for the first example embodiment). Thereafter, the step corresponding to the step shown in FIG. 2d for the first example embodiment is performed.

Claims

1. A method of processing speech, comprising: providing a first set of audio data having audio features in a first bandwidth; down-sampling the first set of audio data to a second bandwidth lower than the first bandwidth; producing, by a high frequency reconstruction network, HFRN, an estimate of audio features in the first bandwidth for the first set of audio data, based on at least the down-sampled audio data of the second bandwidth; inputting, into the HFRN, a second set of audio data having audio features in the second bandwidth; producing, by the HFRN, an estimate of audio features in the first bandwidth for the second set of audio data, based on the second set of audio data having audio features in the second bandwidth, wherein the estimate of audio features in the first bandwidth for the second set of audio data is produced additionally based on meta data embedding for the second set of audio data, wherein the meta data embedding for the second set of audio data includes meta data characterizing a Long Term Average Speech Spectrum, LTASS-based equalization transfer function, wherein the LTASS-based equalization transfer function is applied to produce the estimate of audio features in the first bandwidth for the second set of audio data, and wherein the LTASS-based equalization transfer function maps a spectral tilt of the first set of audio data in the first bandwidth to a spectral tilt of the second set of audio data in the second bandwidth; multiplying the LTASS-based equalization transfer function with at least one perturbation vector to produce at least one new equalization transfer function; and training a Speech Processing System, SPS, using i) the estimate of audio features in the first bandwidth for the first set of audio data, and ii) the estimate of audio features in the first bandwidth for the second set of audio data.

2. The method of claim 1, wherein: the estimate of audio features in the first bandwidth for the first set of audio data is produced additionally based on at least one of i) text transcription of the first set of audio data, and ii) meta data embedding for the first set of audio data.

3. The method of claim 2, wherein: the meta data embedding for the first set of audio data includes meta data characterizing at least one of a speaker, audio environment, and Long Term Average Speech Spectrum, LTASS-based equalization transfer function.

4. The method of claim 2, wherein: the estimate of audio features in the first bandwidth for the second set of audio data is produced additionally based on text transcription of the second set of audio data.

5. The method of claim 4, wherein: the meta data embedding for the second set of audio data includes meta data characterizing at least one of a speaker and audio environment.

6. The method of claim 4, wherein the SPS is an Automatic Speech Recognition, ASR, system, the method further comprising: initially training the ASR system with the first set of audio data having audio features in the first bandwidth.

7. The method of claim 6, further comprising: training the HFRN to take into account at least one of ASR loss and reconstruction loss; wherein the ASR loss refers to a variance between an output produced by the ASR using selected data in the first set of audio data having audio features in the first bandwidth and an output produced by the ASR using the estimate of audio features in the first bandwidth for the selected data in the first set of audio data; and wherein the reconstruction loss refers to a variance between audio features of the selected data and the estimate of audio features for the selected data.

8. The method of claim 7, further comprising: training a plurality of HFRNs with different cut-off frequencies to enable the ASR system to process audio data in multiple bandwidths.

9. A system for performing speech processing, wherein a first set of audio data having audio features in a first bandwidth is provided, and wherein the first set of audio data is down-sampled to a second bandwidth lower than the first bandwidth, the system comprising: a High Frequency Reconstruction Network, HFRN, configured to: a) provide an estimate of audio features in the first bandwidth for the first set of audio data, based on at least the down-sampled audio data of the second bandwidth; b) receive a second set of audio data having audio features in the second bandwidth; c) produce an estimate of audio features in the first bandwidth for the second set of audio data, based on the second set of audio data having audio features in the second bandwidth, wherein the estimate of audio features in the first bandwidth for the second set of audio data is produced additionally based on meta data embedding for the second set of audio data wherein the meta data embedding for the second set of audio data includes meta data characterizing a Long Term Average Speech Spectrum ,LTASS-based equalization transfer function, wherein the LTASS-based equalization transfer function is applied to produce the estimate of audio features in the first bandwidth for the second set of audio data, and wherein the LTASS-based equalization transfer function maps a spectral tilt of the first set of data in the first bandwidth to a spectral tilt of the second set of data in the second bandwidth; and d) multiplying the LTASS-based equalization transfer function with at least one perturbation vector to produce at least one new equalization transfer function; and a Speech Processing System, SPS, trained using i) the estimate of audio features in the first bandwidth for the first set of audio data, and ii) the estimate of audio features in the first bandwidth for the second set of audio data.

10. The system of claim 9, wherein: the estimate of audio features in the first bandwidth for the first set of audio data is produced additionally based on at least one of i) text transcription of the first set of audio data, and ii) meta data embedding for the first set of audio data.

11. The system of claim 10, wherein: the meta data embedding for the first set of audio data includes meta data characterizing at least one of a speaker, audio environment, and Long Term Average Speech Spectrum, LTASS-based equalization transfer function.

12. The system of claim 10, wherein: the estimate of audio features in the first bandwidth for the second set of audio data is produced additionally based on text transcription of the second set of audio data.

13. The system of claim 12, wherein: the meta data embedding for the second set of audio data includes meta data characterizing at least one of a speaker and audio environment.