Speech restoration with improved intelligibility

Abstract

The disclosure relates to a computer-implemented method of training a neural network-based language model for restoring clean speech from a distorted speech waveform. The method includes obtaining an input time-frequency representation of the distorted speech waveform; determining, from the input time-frequency representation, one or more audio features (e.g., semantic representations) associated with the distorted speech waveform using a neural network-based multi-stage speech encoder; determining conditioning information for the language model based at least in part on the one or more audio features; and jointly training the language model and the speech encoder, wherein the language model is trained based at least in part on the conditioning information.

Description

[0001] SPEECH RESTORATION WITH IMPROVED INTELLIGIBILITY

[0002] Cross-Reference to Related Applications

[0003] This application claims the benefit of priority from International Patent Application No. PCT / CN2024 / 080815, filed March 8, 2024, U.S. Provisional Application Serial No. 63 / 664,566, filed June 26, 2024, International Patent Application No. PCT / CN2024 / 116230, filed September 2, 2024, Spanish Patent Application No. P202430693, filed September 2, 2024, U.S. Provisional Application Serial No. 63 / 722,192, filed November 19, 2024, and U.S. Provisional Application Serial No. 63 / 758,701, filed February 14, 2025, each of which is incorporated by reference herein in its entirety.

[0004] Technical Field

[0005] The present disclosure relates to speech restoration and, in particular, to methods and systems for speech restoration by training a speech encoder in conjunction with a machine learning model to provide enhanced quality of restored speech.

[0006] Background

[0007] Speech restoration refers to the task of providing high-quality speech from degraded speech, and especially aims at restoring full-band speech with enhanced quality and intelligibility from a corrupted signal. Unlike conventional speech enhancement, which typically refers only to speech denoising and dereverberation, there are many more types of degradations to be considered for speech restoration, such as noise, reverberation, low-bandwidth, clipping, packet loss, etc. In particular, speech restoration addresses a diverse set of tasks including those that are generative in nature, such as bandwidth extension, packet loss concealment, and so on. Existing methods for speech restoration include adversarial learning and diffusionbased approaches. More recently, language models have also been used for speech enhancement, which, however, only takes in to account speech denoising.

[0008] In general, it has been shown to be much easier to train language models at scale compared with adversarial learning. However, the use of some existing language models may require a significant number of sampling steps during inference process, which leads to extremely high computational load for a speech restoration system. Besides, existing models (or neural networks) may introduce artifacts during training / inference process, which causes distortions to the processed speech including spectral holes, missing high frequency components, changing the speaker voice, cutting off part of a word, etc. Such distortions caused by neural networks have been problematic for speech enhancement, despite that neural networks can successfully remove noise. Conventionally, this problem may be tackled by involving design of loss functions to penalize over suppressing speech during training.

[0009] A further problem is that the improved perceptual quality after processing using existing models may not translate to improved intelligibility (which can typically be measured by the automatic speech recognition word error rate (WER) of automatic speech recognition (ASR) systems), since removing distortions may undesirably alter the phonetic content of speech.

[0010] There is thus a need for developing improved techniques for methods and systems of speech restoration which can remove the above mentioned distortions in speech in a computationefficient and reliable manner. Especially, it is desirable to develop a deep generative approach that effectively eliminates distortions / artifacts introduced by neural network models during the restoration process with reduced computational efforts. It is further desirable to improve the intelligibility of the generated speech.

[0011] Summary

[0012] In view of this need, the present disclosure provides computer-implemented methods and corresponding systems for restoration of high-quality speech from all types of distortions, as well as computer programs and computer-readable storage media, having the features of the respective independent claims.

[0013] In particular, the present disclosure proposes to jointly train a speech encoder and a machine learning model (e.g., a neural network-based language model such as a masked language model, or an autoregressive language model) for speech restoration. It is appreciated that a masked language model based on neural network processing may require much fewer sampling steps during inference. To achieve efficient speech restoration, the present disclosure further proposes to employ a fusion network designed to effectively bridge the speech encoder and the language model for both training and inference processes. Specifically, the proposed fusion methods may include, for example, concatenating one or more (or all) intermediate features of the speech encoder, or performing a learned weighted sum based on attention. In addition, the present disclosure also proposes to apply classifier free guidance (e.g., at the inference stage) to effectively reduce distortions to generated clean speech caused by neural network processing (e.g., during the training stage).

[0014] One aspect of the present disclosure relates to a computer-implemented method for speech restoration, especially for training a neural network-based language model in a speech restoration system for generating / restoring clean speech from a distorted speech waveform, i.e., at a training stage of the speech restoration system. The method may include obtaining an input time-frequency (also referred to as “spectral-temporal”) representation of the distorted speech waveform. Such input time-frequency representation may, for example, relate to local and global spectral and temporal speech patterns of the distorted speech waveform. The method may further include determining one or more audio features (which may relate to a plurality of intermediate speech representations of the input time-frequency representation e.g., semantic representations) associated with the distorted speech waveform. In particular, the one or more audio features may be determined from the input time- frequency representation using e.g., a neural network-based multi-stage speech encoder. Also, the method may include determining conditioning information for the language model based at least in part on the one or more audio features. For example, the one or more audio features (or the plurality of intermediate speech representations) may be combined (or fused) to provide a feature condition (i.e., the conditioning information) to the neural network language model. In addition, the method may also include jointly training the language model and the speech encoder. Notably, the language model may be trained based at least in part on the conditioning information.

[0015] Configured as above, by combining the audio features obtained from the different stages / levels of the speech encoder, optimum speech representation can be provided for the condition determination, which further improves the accuracy of training the speech restoration system. By simultaneously training the language model and the speech encoder, different types of distortions may be jointly and effectively removed from the audio data, allowing for efficient generation of high-quality speech.

[0016] In some embodiments, the method may further take into account semantic information within the speech for training the speech restoration system. Specifically, the method may also include obtaining a reference time- frequency representation of a ground truth speech waveform for the distorted speech waveform. The method may additionally include determining one or more semantic representations associated with the distorted speech waveform based on the one or more audio features. The method may also include determining one or more reference semantic representations associated with the ground truth speech waveform. More specifically, the one or more reference semantic representations may be determined from the reference time- frequency representation using a pre-trained auxiliary model (e.g., a teacher model). Accordingly, the speech encoder may be trained based on the one or more semantic representations and the one or more reference semantic representations, using an auxiliary loss function.

[0017] In some embodiments, determining the one or more semantic representations from the one or more audio features may include applying a fully connected pooling layer. In some embodiments, the one or more semantic representations may include one or more pseudo- phonetic labels. In particular, the one or more reference semantic representations may include one or more reference pseudo-phonetic labels. In some embodiments, the pre-trained auxiliary model may include a self-supervised model. For example, the pre-trained auxiliary model may be a pre-trained HuBERT (Hidden-Unit Bidirectional Encoder Representations from Transformers) model.

[0018] Configured as above, by taking into account the semantic information (e.g., the one or more semantic representations) of the speech, both perceptual quality and intelligibility of the generated / processed speech can be improved in an effective manner, thereby enhancing the performance of the speech restoration system. In particular, the proposed method allows for providing the optimum trade-off between intelligibility and quality of the speech for full-band speech restoration.

[0019] In some embodiments, the method may further include applying a multi-layer perceptron (e.g., at the speech encoder) to generate a predicted time-frequency representation of a clean speech waveform. In some embodiments, the method may also include obtaining a reference time-frequency representation of a ground truth speech waveform for the distorted speech waveform. In particular, the speech encoder may be trained based on the reference timefrequency representation and the predicted time-frequency representation predicted by the speech encoder. In some embodiments, the speech encoder may be trained using a regression loss. For example, training the speech encoder may include minimizing a difference between the predicted time-frequency representation and the reference time-frequency representation by adapting one or more parameters of the multi-stage speech encoder based on a regression loss function.

[0020] In some embodiments, the one or more audio features may be determined based on a plurality of intermediate speech representations (which may relate to the one or more audio features) of the input time-frequency representation generated within the speech encoder. In some embodiments, the plurality of intermediate speech representations may be generated by applying a plurality of successive transformer layers (which may be provided e.g., at the speech encoder). Specifically, each transformer layer may generate a respective intermediate speech representation.

[0021] For determining / generating the plurality of intermediate speech representations, the method may further include transforming (e.g., by a second multi-layer perceptron, which may be provided e.g., at the speech encoder) a frame size of the input time- frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers. Besides, the method may also include combining a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model.

[0022] In some embodiments, the further intermediate speech representation and the plurality of intermediate speech representations may be combined by concatenating one or more features from the second multi-layer perceptron and the respective transformer layers as a feature vector and applying a learnable projection to the feature vector using a linear layer.

[0023] In some embodiments, the further intermediate speech representation and the plurality of intermediate speech representations may be combined by summing one or more features from the second multi-layer perceptron and the respective transformer layers with a corresponding weight coefficient determined by applying an attention algorithm.

[0024] In some embodiments, the method may further include randomly providing the conditioning information (e.g., the above mentioned feature condition associated with the combined intermediate speech representations) to the language model for conditioned training of the language model (i.e., for training the neural network model with condition). The method may also include replacing a portion of the conditioning information with one or more learnable embedding parameters as unconditional input for training the language model without condition.

[0025] Accordingly, training the language model may be based further on the unconditional input comprising the one or more learnable embedding parameters. For example, a number of training cycles for conditioned training of the language model may be less than a total number of training cycles for training of the language model. More specifically, a number of training cycles for training the language model without condition may be given by a predetermined ratio. For example, a duration of training the language model without condition may be 10% of a total duration for training the language model.

[0026] In some embodiments, the method may include tokenizing a reference waveform associated with a ground truth speech signal to convert the reference waveform into a codegram of a predefined size containing discrete tokens for training the language model. In some embodiments, the language model may be trained based on a cross-entropy loss function. In some embodiments, the language model may include an autoregressive language model, or the language model may include a masked language model.

[0027] In some embodiments where a masked language model is implemented as the neural network model, the method may include randomly masking one or more discrete tokens in the codegram to generate a partially-masked codegram, and converting the partially-masked codegram to a plurality of embedding vectors. The method may also include summing the plurality of embedding vectors, and providing a summation of the summed embedding vectors with the conditioning information as training input to the masked language model.

[0028] More specifically, the language model may contain a plurality of transformer blocks followed by a multi-layer perceptron. Accordingly, for training the language model, the method may further include extracting a plurality of masked tokens from the ground truth speech signal. The method may also include estimating, by the multi-layer perceptron, a plurality of predicted indices corresponding to the masked tokens. Moreover, the method may include determining a difference between the predicted indices and actual indices of the masked tokens. Also, the method may include minimizing the difference between the predicted indices and the actual indices of the masked tokens based on a cross-entropy loss function. Configured as above, by designing and employing a fusion network (by means of e.g., concatenating all intermediate features of the speech encoder, or performing a learned weighted sum based on attention), the speech encoder and the language model can be effectively bridged for both training and inference processes, thereby achieving efficient speech restoration with enhanced performance. It is further appreciated that, by jointly optimizing the loss functions (e.g., cross-entropy loss and / or regression loss) to train the language model and the speech encoder, training of the speech restoration system may be conducted in a computation-efficient and reliable manner. The feasibility of the methods as proposed in the present disclosure is further appreciated, as the proposed methods may be cascaded (incorporated) with existing models to fix the artifacts introduced by the existing models.

[0029] One aspect of the present disclosure relates to a computer-implemented method for speech restoration, especially for using a neural network-based language model for restoring clean speech from a distorted speech waveform, i.e., at an inference stage of the speech restoration system. Similar to the proposed training methods as described above, the method may include obtaining an input time-frequency (also referred to as “spectral-temporal”) representation of the distorted speech waveform. The method may further include determining one or more audio features (which may relate to a plurality of intermediate speech representations of the input time-frequency representation e.g., semantic representations) associated with the distorted speech waveform. In particular, the one or more audio features may be determined from the input time-frequency representation using a neural network-based multi-stage speech encoder. Also, the method may include determining conditioning information for the language model based at least in part on the one or more audio features. For example, the one or more audio features (or the plurality of intermediate speech representations) may be combined (or fused) to provide a feature condition (i.e., the conditioning information) to the neural network mode, which is similar to the proposed training methods as described above.

[0030] In addition, the method may also include generating a plurality of discrete tokens of a codegram corresponding to a clean speech signal based at least in part on the conditioning information. Notably, the plurality of discrete tokens of a codegram may be generated from a fully-masked codegram containing a plurality of masked tokens. For example, the plurality of discrete tokens may be generated from a fully-masked codegram containing exclusively a plurality of masked tokens. Furthermore, the method may include generating a restored speech waveform associated with the clean speech signal from the plurality of discrete tokens of the codegram.

[0031] Configured as above, by combining the audio features obtained from the different stages / levels of the speech encoder, optimum speech representation can be provided for the condition determination, which further improves the accuracy of training the speech restoration system. In this way, different types of distortions can be removes from speech in a computation-efficient and reliable manner. In other words, distortions / artifacts introduced by neural network models can be effectively eliminated during the restoration process with reduced computational efforts.

[0032] In some embodiments, the plurality of discrete tokens of the codegram may be generated based on an iterative sampling procedure. More specifically, the method may apply the classifier free guidance (CFG) approach for the iterative sampling procedure for the speech restoration.

[0033] In detail, the iterative sampling procedure according to the present disclosure may include, for each iteration, computing a first logit of first predicted tokens generated from the language model using the conditioning information and, subsequently, computing a second logit of second predicted tokens generated from the language model without the conditioning information, or vice versa. The iterative sampling procedure may also include combining the first logit and the second logit using a controllable parameter to obtain a logit with guidance. In particular, the controllable parameter may be indicative of a degree of attachment of the generation of the restored speech waveform associated with the clean speech signal to the conditioning information. Besides, the iterative sampling procedure may then also include sampling one or more output tokens of the language model generated from a previous iteration for re-masking based on the logit with guidance.

[0034] In some embodiments, the language model may include an autoregressive language model. In some embodiments, the language model may include a masked language model. In the latter case, the method may further include obtaining the fully-masked codegram containing a plurality of masked tokens (e.g., obtaining the fully-masked codegram containing exclusively a plurality of masked tokens). Also, the method may include converting the fully-masked codegram to a plurality of embedding vectors. The method may further include summing the plurality of embedding vectors, and providing, based on the logit with guidance, a summation of the summed embedding vectors with the conditioning information as input to the masked language model. Accordingly, the method may include predicting, by the language model, the plurality of masked tokens for predicting the plurality of discrete tokens of the codegram.

[0035] In some embodiments, the iterative sampling procedure may further include determining a predicted accuracy and a predicted gain for each predicted masked token. In particular, predicting the plurality of discrete tokens may include retaining a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is greater than or equal to a predefined value. Alternatively, predicting the plurality of discrete tokens may include discarding a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is smaller than a predefined value, and may also include re-predicting the discarded masked token by the masked language model.

[0036] In some embodiments, re-predicting the discarded masked token may include updating the fully-masked codegram based on the retained predicted masked token and the discarded masked token, converting the updated fully-masked codegram to a plurality of updated embedding vectors, obtaining the logit with guidance based on an output of the masked language model without the conditioning information, summing the plurality of updated embedding vectors, and providing, based on the logit with guidance, a summation of the summed updated embedding vectors with the conditioning information as updated input to the masked language model for prediction.

[0037] The proposed method for operating the speech restoration system at the inference stage may also include processing steps similar to the proposed training method (for operating at the training stage) as described above. In some embodiments, the one or more audio features may be determined based on a plurality of intermediate speech representations of the input timefrequency representation generated within the speech encoder. In some embodiments, the plurality of intermediate speech representations may be generated by applying a plurality of successive transformer layers each generating a respective intermediate speech representation. For determining the plurality of intermediate speech representations, the method may include transforming, by a second multi-layer perceptron, a frame size of the input time-frequency representation to a transformer layer size for passing the input timefrequency representation to the plurality of successive transformer layers. In some embodiments, the method may further include combining a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model. For example, the further intermediate speech representation and the plurality of intermediate speech representations may be combined by concatenating one or more features from the second multi-layer perceptron and the respective transformer layers as a feature vector and applying a learnable projection to the feature vector using a linear layer. Alternatively, the further intermediate speech representation and the plurality of intermediate speech representations may be combined by summing one or more features from the further multi-layer perceptron and the respective transformer layers with a corresponding weight coefficient determined by applying an attention algorithm.

[0038] Configured as above, by applying classifier free guidance during the inference stage to adjust (e.g., via a controllable parameter provided by the CFG) the degree of attachment of the generation process to the mixture, different types of distortions (such as bandwidth extension, spectral hole inpainting, packet loss concealment, etc.) to generated clean speech caused by the neural network processing can be effectively reduced, which enhances the performance of the speech restoration system in a computation-efficient manner.

[0039] One aspect of the present disclosure relates to a system for speech restoration, especially for training (e.g., at a training stage) and using (e.g., at an inference stage) a neural networkbased language model in a speech restoration system for restoring clean speech from a distorted speech waveform. In particular, the proposed system may be implemented for conducting the proposed training method and / or the proposed inference method as described above.

[0040] The system may include a neural network-based multi-stage speech encoding unit configured to obtain an input time-frequency representation of a distorted speech waveform, and determine, from the input time-frequency representation, one or more audio features associated with the distorted speech waveform. The system may further include a neural network-based language model configured to generate a plurality of discrete tokens of a codegram corresponding to a clean speech signal. The system may also include a decoding unit configured to generate a restored speech waveform associated with the clean speech signal from the plurality of discrete tokens of the codegram. Moreover, the system may include a fusion element configured to generate conditioning information for the language model based on the one or more audio features. In particular, the plurality of discrete tokens of the codegram may be generated based at least in part on the conditioning information.

[0041] In some embodiments, the speech encoding unit may include a multi-layer perceptron configured to generate a predicted time-frequency representation of a clean speech waveform based on a plurality of intermediate speech representations of the input time-frequency representation. Moreover, the speech encoding unit may be further configured to determine one or more semantic representations based on the one or more audio features. In some embodiments, the speech encoding unit may further include a plurality of successive transformer layers each configured to generate a respective intermediate speech representation. In some embodiments, the speech encoding unit may further include a second multi-layer perceptron configured to transform a frame size of the input time- frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers.

[0042] In some embodiments, the fusion element may be configured to combine a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model. In some embodiment, the system may further include a tokenization unit configured to tokenize a reference waveform associated with a ground truth speech signal to convert the reference waveform into a reference codegram of a predefined size containing reference discrete tokens for training the language model.

[0043] In some embodiments, the speech encoding unit and the language model may be jointly trained based on a regression loss function and / or a cross-entropy function. In some embodiments, the language model may include an autoregressive language model, or a masked language model.

[0044] In some embodiments where the language model includes a masked language model, the system may further include a masking unit configured to randomly mask one or more reference discrete tokens in the reference codegram to generate a partially-masked codegram, an embedding unit configured to convert the partially -masked codegram to a plurality of embedding vectors, and a summing unit configured to sum the plurality of embedding vectors and to provide a summation of the summed embedding vectors with the conditioning information as training input to the masked language model.

[0045] In some embodiments, the masked language model may include a plurality of transformer blocks followed by a multi-layer perceptron configured to, for training the masked language model, extract a plurality of masked tokens from the ground truth speech signal, estimate a plurality of predicted indices corresponding to the masked tokens, determine a difference between the predicted indices and actual indices of the masked tokens, and minimize the difference between the predicted indices and the actual indices of the masked tokens based on a cross-entropy loss function.

[0046] In some embodiments, the system may further include a logit estimation unit configured to compute a first logit of first predicted tokens generated from the language model using the conditioning information and, subsequently, compute a second logit of second predicted tokens generated from the language model without the conditioning information, or vice versa. Besides, the logit estimation unit may be further configured to combine the first logit and the second logit using a controllable parameter to obtain a logit with guidance. In particular, the controllable parameter may indicate of a degree of attachment of the generation of the restored speech waveform associated with the clean speech signal to the conditioning information. Furthermore, the plurality of discrete tokens of the codegram may be generated based on an iterative sampling procedure by classifier free guidance (CFG). Specifically, the iterative sampling procedure may include sampling, for each iteration, one or more output tokens of the language model generated from a previous iteration for remasking based on the logit with guidance.

[0047] Configured as above, the proposed system can perform efficient speech restoration and allows for generation of enhanced-quality speech in the presence of noise and different types of distortions. It is also appreciated that the proposed system may be used to perform the processing steps of the training and inference methods as described above.

[0048] In view of the above, the present disclosure proposes improved techniques for methods and systems of speech restoration which removes the speech distortions in a computationefficient and reliable manner. In particular, the present disclosure proposes a deep generative approach that eliminates distortions / artifacts introduced by neural network models during the restoration process with reduced computational efforts. According to another aspect, an apparatus for speech restoration is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform all steps of the methods according to preceding aspects and their embodiments.

[0049] According to a further aspect, a computer program is described. The computer program may comprise executable instructions for performing the methods or method steps outlined throughout the present disclosure when executed by a computing device (e.g., processor).

[0050] According to another aspect, a computer-readable storage medium is described. The storage medium may store a computer program adapted for execution on a computing device (e.g., processor) and for performing the methods or method steps outlined throughout the present disclosure when carried out on the computing device.

[0051] It should be noted that the methods and systems including its preferred embodiments as outlined in the present disclosure may be used stand-alone or in combination with the other methods and systems disclosed in this document. Furthermore, all aspects of the methods and systems outlined in the present disclosure may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.

[0052] It will be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, the details of the disclosed method(s) can be realized by the corresponding apparatus, and vice versa, as the skilled person will appreciate. Moreover, any of the above statements made with respect to the method(s) (and, e.g., their steps) are understood to likewise apply to the corresponding apparatus (and, e.g., their blocks, stages, units), and vice versa.

[0053] Brief Description of the Drawings

[0054] The invention is explained below in an exemplary manner with reference to the accompanying drawings, wherein

[0055] Fig. 1 schematically illustrates an example apparatus / system for speech restoration during a training stage in accordance with embodiments of the present disclosure;

[0056] Fig. 2 schematically illustrates a detailed implementation of the example apparatus / system of Fig. 1 for speech restoration in accordance with embodiments of the present disclosure; Fig. 3 schematically illustrates another example apparatus / system for speech restoration taking into account semantic information in accordance with embodiments of the present disclosure;

[0057] Fig. 4 schematically illustrates an example apparatus / system for speech restoration during an inference stage in accordance with embodiments of the present disclosure;

[0058] Fig. 5 schematically illustrates a detailed implementation of the example apparatus / system of Fig. 4 for speech restoration in accordance with embodiments of the present disclosure;

[0059] Fig. 6 shows an example flowchart of a method 600 for speech restoration during a training stage in accordance with embodiments of the disclosure;

[0060] Fig. 7 shows an example flowchart of a method 700 for speech restoration during an inference stage in accordance with embodiments of the disclosure; and

[0061] Fig. 8 schematically illustrates an example apparatus for implementing the proposed speech restoration methods in accordance with embodiments of the disclosure.

[0062] Detailed Description

[0063] In the following, example embodiments of the disclosure will be described with reference to the appended figures. Identical elements in the figures may be indicated by identical reference numbers, and repeated description thereof may be omitted.

[0064] Fig. 1 schematically illustrates an example apparatus / system for speech restoration in accordance with embodiments of the present disclosure. In particular, the apparatus / system 100 may be operated during a training stage of the speech restoration process, for example, for training a neural network-based language model to restore clean speech from a distorted speech waveform.

[0065] The system 100 (as a speech restoration system) may be or may be part of a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or a cluster of computers), or a distributed system between a computing device and a server which is suitable for conducting the speech restoration process. In detail, the speech restoration system 100 comprises a speech encoding unit 10, e.g., a neural network-based multi-stage speech encoder, which may be configured to receive or obtain a distorted speech waveform, a machine learning model 11 such as a neural network-based language model, which is to be trained for speech restoration, and a fusion element 12 (as a fusion network) coupled between the multi-stage speech encoding unit 10 and the neural network-based language model 11.

[0066] Specifically, the speech encoding unit 10 may be configured to obtain an input timefrequency (temporal-spectral) representation 1 of the distorted speech waveform. In some embodiments, the speech encoding unit 10 may take an artificially created distorted speech wave and convert it to a spectrogram (e.g., by means of short-time Fourier transform, STFT). For example, the speech encoding unit 10 may take an artificially created distorted speech waveform at a sampling rate of 44. 1 kHz and convert it into a 1025-dimensional magnitude spectrogram with a frame length of 2048 points, and a hop size of 512 points (the size of STFT frames). It is noted that the aforementioned example sampling rates and spectrogram dimensions are non-limiting examples for obtaining the input time-frequency representation of the distorted speech waveform as explicitly illustrated in the present disclosure, and other sampling rates and spectrogram dimensions are feasible and within the scope of the present disclosure.

[0067] Furthermore, the speech encoding unit 10 may also determine one or more audio features 2-1, 2-2, ..., 2-N (e.g., semantic representations, or other non-semantic representations) associated with the distorted speech waveform for training the speech restoration system 100. The one or more audio features 2- 1 , 2-2, . . . , 2-N may be determined from the input time-frequency representation 1 obtained by the speech encoding unit 10. The determined one or more audio features 2-1, 2-2, . . ., 2-N may then be provided to the fusion element 12 which may subsequently determine conditioning information 4 for the neural network-based language model 11 based (at least in part) on the one or more audio features 2- 1 , 2-2, . . . , 2-N.

[0068] During the training stage of the speech restoration system 100, the speech encoding unit 10 and the neural network-based language model 11 may be jointly trained (by means of e.g., a regression loss and / or a cross-entropy loss) to restore high-quality noise-reduced speech from the distorted speech waveform (which may include different types of distortions, e.g., spectral holes, missing high frequency components, changing the speaker voice, cutting off part of a word, etc.). In particular, the neural network-based language model 11 may be trained based (at least in part) on the conditioning information 4 as provided by the fusion element 12. Alternatively, the speech encoding unit 10 and the neural network-based language model 11 may also be trained separately (e.g., based on the respective loss functions). Specifically, the conditioning information 4 may be randomly provided to the neural network-based language model 11 for conditioned training of the language model 11. In addition, a portion of the conditioning information 4 may be replaced with one or more learnable embedding parameters (e.g., by means of embedding tables which may be provided in the language model 11) as unconditional input for training the language model 11 without condition. Accordingly, the neural network-based language model 11 may be trained based further on the unconditional input comprising the one or more learnable embedding parameters.

[0069] In other words, training of the language model 11 may include conditioned training and training without condition. As an implementation example, a number of training cycles for conditioned training of the language model 11 is less than a total number of training cycles for training of the language model 11. In particular, the ratio of the number of training cycles for conditioned training of the language model 11 to the total number of training cycles may correspond to a predetermined ratio, or a number of training cycles for training the language model 11 without condition is given by a predetermined ratio.

[0070] It is appreciated that the neural network-based language model 11 may take discrete tokens as their input / output, such as words in natural language processing. Therefore, to take advantage of the language model 11 , audio tokenization may be additionally conducted by the speech restoration system 100. As a preferred embodiment according to the present disclosure, the speech restoration system 100 may further comprise a tokenizing unit 13 configured to tokenize a reference waveform associated with a ground truth speech signal 3.

[0071] Specifically, the tokenizing unit 13 may obtain / receive the reference waveform associated with the ground truth speech signal 3 and convert the reference waveform into a codegram 5 of a predefined size containing discrete tokens for training the language model 11. A size of a codegram 5 may be represented by KxT, where K is the number of codebooks and T is the number of frames determined based on the tokenizing unit 13. Such a tokenizing unit 13 may be implemented by a pre-trained neural codec (which can perform both encoding and decoding functions), for example. However, other types of tokenizers and / or other methods for audio tokenization are feasible and within the scope of the present disclosure.

[0072] Preferably, a masked language model may be used for implementing the neural networkbased language model 11, which is to be described in Fig. 2 as below. Besides, a cross- entropy loss function may be applied to train the neural network-based language model 11 (e.g., a masked language model) as described in Fig. 2 below.

[0073] Fig. 2 schematically illustrates an example apparatus / system for speech restoration in accordance with embodiments of the present disclosure. Similar to the system architecture shown in Fig. 1 (i.e., the speech restoration system 100), the speech restoration system 100a may be or may be part of a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or a cluster of computers), or a distributed system between a computing device and a server which is suitable for conducting the speech restoration process, and may be operated during a training stage of the speech restoration process, for example, for training a neural network-based language model (e.g., a masked language model) to restore clean speech from a distorted speech waveform. Also, the speech restoration system 100a may comprise components similar to the speech restoration system 100, such as a speech encoding unit 10, a neural network-based language model 1 1, a fusion element 12 and a tokenizing unit 13 which perform similar respective functions as the above described speech restoration system 100. The details of these components of the speech restoration system 100a similar to those in the speech restoration system 100 will not be repeated herein for conciseness.

[0074] The speech restoration system 100a may be regarded as an example implementation of the speech restoration system 100. More specifically, the speech encoding unit 10 of the speech restoration system 100a may be further configured to generate a plurality of intermediate speech representations (now shown) of (or associated with) the input time-frequency representation 1 for determining the one or more audio features. 2-1, 2-2, . .., 2-N. As an implementation example of generating the intermediate speech representations, the speech encoding unit 10 of the speech restoration system 100a may be configured to apply a plurality of successive transformer layers (or blocks)101 (e.g., three transformer layers 101-1, 101-2, 101-3), and each transformer layer may generate a respective intermediate speech representation. Moreover, the speech encoding unit 10 of the speech restoration system 100a may also be configured to determine a further intermediate speech representation (of the plurality of intermediate speech representations) by transforming a frame size of the input time-frequency representation 1 to a transformer layer size via a multi-layer perceptron (MLP) 102-1 for passing the input time- frequency representation 1 to the plurality of successive transformer layers 101. Thus, the intermediate speech representation from the respective transformer layer 101 and the further intermediate speech representation from the multi-layer perceptron 102-1 may then be provided to the fusion element 12 for determining the conditioning information 4. For example, the conditioning information 4 may be determined by combining the further intermediate speech representation from the multi-layer perceptron 102-1 and the plurality of intermediate speech representations from the plurality of successive transformer layers 101, which may then be used for training the language model 11 as described above.

[0075] As an example of implementing the speech encoding unit 10 of the speech restoration system 100a, the multi-layer perceptron (MLP) 102-1 may contain two affine layers with a GELU (Gaussian Error Linear Unit) nonlinear activation function in between, which transforms each 1025-dim STFT frame to D dimension (D=512 as an example). Subsequently, the D- dimensional speech representation may go through a stack of N multi-head self-attention transformer blocks 101 (N=6 as an example), which models the local and global spectral and temporal speech patterns. The skilled person may appreciate that other spectrogram dimensions other number of transformer blocks / layers are likewise applicable and within the scope of the present disclosure.

[0076] In addition, the speech encoding unit 10 of the speech restoration system 100a may be configured to further obtain a reference time- frequency representation of a ground truth speech waveform 1 ’ (e.g., ground truth speech STFT) for the distorted speech waveform. Also, the speech encoding unit 10 may be configured to further apply another (a second) multi-layer perceptron 102-2 to generate a predicted time-frequency representation of a clean speech waveform (e.g., predicted clean STFT). Accordingly, the speech encoder 10 of the speech restoration system 100a may be trained based on the reference time-frequency representation and the predicted time- frequency representation predicted by the speech encoder 10, using e.g., a regression loss 103. For example, the speech encoding unit 10 may be trained by minimizing a difference between the predicted time-frequency representation and the reference time-frequency representation. The difference may be minimized by e.g., adapting one or more parameters of the speech encoding unit 10 based on a regression loss function.

[0077] As an example of implementing the speech encoding unit 10 of the speech restoration system 100a, another multi-layer perceptron (MLP) 102-2 may map the D-dimensional output features from the last transformer block back to 1025-dim (e.g., the dimension of the input spectrogram) to generate the predicted magnitude spectrogram (e.g., predicted clean STFT). During training, given the ground truth speech 1 ’, an arbitrary regression loss function may be used to minimize the distance between the predicted clean speech STFT and the ground truth speech STFT. Such examples may include simple LI or L2 losses, or other more advanced losses, such as the multi-STFT loss. Additionally, the STFT may be converted back to the waveform domain by an inverse STFT to compute the loss in the waveform (time) domain (such as LI, L2, or log / MSE (Mean Squared Error) losses, SI-SNR (Scale-Invariant SignaLto-Noise Ratio) losses, etc.). Furthermore, the STFT may be grouped into a perceptually motivated banded domain to compute the loss defined on the banded energy, such as the mel band.

[0078] It is further appreciated that the speech encoder (e.g., the speech encoding unit 10 of the proposed speech restoration system 100, 100a) may be built upon any architectures, not necessarily transformer blocks. Also, the regression loss for training the speech encoder may be any loss defined in any domain, and the speech encoder may be separately or jointly trained with the language model.

[0079] It is noted that the functionality of the speech encoder (e.g., the speech encoding unit 10 of the proposed speech restoration system 100, 100a) is to preprocess the distorted speech, so that the intermediate speech representations become cleaner than the raw input. This can be achieved by using e.g., a standalone speech encoder to perform speech denoising / dereverberation tasks. However, merely training a speech encoder with a regression loss (in an arbitrary domain) may not be able to effectively perform generative speech restoration tasks, such as bandwidth extension, spectral hole inpainting, packet loss concealment, etc. Therefore, by training the speech encoder only as described above, the proposed speech restoration system may preprocess the noisy and distorted speech for the restoration process, and, in order to deliver a final output (restored clean speech), a further methodology is developed by the present disclosure to enhance the ability of the proposed speech restoration system.

[0080] It is further noted that the fusion element (as a fusion network) 12 may serve as a bridge between the speech encoder 10 and the language model 11. In other words, it may fuse the intermediate speech representations from all the transformer blocks 101 as well as from the initial MLP 102-1, and send the fused features to the language model 11 for further processing. It may be possible for the fusion network to take the feature(s) only from the last transformer block. However, since the speech encoder may cause distortions (such as oversuppressing clean speech) as it removes noise from the noisy input, the last transformer layer may not provide the best speech representation. Therefore, two example methods of implementing the fusion element 12 of the speech restoration system 100a that use features from one or more (or all) transformer blocks 101, including the output from the initial MLP layer 102-1, are proposed as below.

[0081] A. Concatenation: This approach concatenates the features from all the N+l layers, yielding a D*(N+1) dimensional feature vector for each frame. Then, a linear layer projects the D*(N+1) elements to the D dimension. Thus, all the features from the N+l layers are combined by the learnable projection.

[0082] B. Attention: an attention mechanism is designed to learn from the data the relative importance of each layer in the speech encoder. In addition, this relative importance is time and channel dependent on the design. Specifically, from each frame at time t, it contains the features from all the N+l layers in a tensor the following processing is performed: where WZ1and W2are learnable projection matrices, which first project the D-dim feature vector at each frame t from any of the N+l layers to an M dimensional vector (D=512, M=128 as an example), and then projects the resulting feature vector back to D-dim, with a nonlinear function f (such as a ReLU (Rectified Linear Unit) function) in between. After normalizing by the softmax function along the layer dimension, N+l attention weights can be obtained for each frame and each channel (a channel refers to an element in a D-dim feature vector). These attention weights in the tensor sum up to 1 across the N+l layers for each of the D dimensions. Finally, an element-wise multiplication (represented by ©) of he resulting tensor along the layer dimension to obtain the aggregated feature vector for frame t. Therefore, each aggregated feature vector at time t is a weighted sum of all the features from N+ 1 layers at the same frame t, and the weights are learned by the attention algorithm.

[0083] Accordingly, the fusion element 12 of the speech restoration system 100a may thus be configured to combine the further intermediate speech representation and the plurality of intermediate speech representations by concatenating one or more features from the multilayer perceptron 102-1 and the respective transformer layers 101 as a feature vector and applying a learnable projection to the feature vector using a linear layer. Alternatively, the fusion element 12 of the speech restoration system 100a may thus be configured to combine the further intermediate speech representation and the plurality of intermediate speech representations by summing one or more features from the multi-layer perceptron 102-1 and the respective transformer layers 101 with a corresponding weight coefficient determined by applying an attention algorithm.

[0084] It is noted that the aforementioned example speech encoder 10 (speech encoding unit) and fusion element (fusion network) 12 of the proposed speech restoration system 100a are nonlimiting examples for restoring the distorted speech waveform as explicitly illustrated in the present disclosure. Other architectures and approaches for implementing the speech encoder and the fusion network are feasible and within the scope of the present disclosure.

[0085] Fig. 2 further illustrates a detailed example implementation of the neural network-based language model 11 according to an embodiment of the present disclosure. Herein, the language model 11 may comprise a masked language model 114, and a masking module 111 may be provided for generating a training input for the masked language model 114. More specifically, the masking module 111 may be configured to randomly mask one or more discrete tokens in the codegram 5 (output by the tokenizing unit 13 as described above) to generate a partially-masked codegram. Also, one or more embedding tables 112 may be applied to convert the partially-masked codegram to a plurality of embedding vectors which may then be summed. A summation unit 113 may also be provided to sum the (summed) embedding vectors with the conditioning information 4 to provide the training input to the masked language model 114.

[0086] Similar to the encoding unit 10, the masked language model 114 may comprise a plurality of transformer blocks followed by a multi-layer perceptron (not shown). Hence, in addition to the encoding unit 10, the masked language model may also be trained based on natural language processing. Specifically, to train the masked language model 114, a plurality of masked tokens may be extracted from the ground truth speech signal 3, and a plurality of predicted indices corresponding to the masked tokens may be then estimated (e.g., by the multi-layer perceptron within the language model). Subsequently, a difference between the predicted indices and actual indices 6 of the masked tokens may be determined which may then be minimize based on a cross-entropy loss function 14.

[0087] As an example of implementing the language model 11 of the speech restoration system 100a, a MaskGIT (Masked Generative Image Transformer) as the backbone algorithm of the language model may be employed. Specifically, during training, a ground truth clean speech 3 may be converted (by the tokenizing unit 13, such as a Descript Audio Codec (DAC) encoder / decoder as described below) to a codegram 5 of size 9xT, and a (randomly selected) percentage of the tokens in the (DAC) codegram 5 may be randomly masked by replacing the tokens to be masked by a special token index 1024 (since the original token indices range from 0 to 1023). 9 learnable embedding tables (each with 1025 entries) 112 may then be used to convert the partially masked codegram to 9 tensors, each representing 1 codebook. Each tensor size is TxD, where T is the number of frames, and D is the dimension of the transformer layers. Besides, each D-dim feature vector in a row of this tensor may embed a token from a codebook at frame t, and the embeddings may be summed at the same time instance from all the 9 tensors, resulting in a tensor of size TxD, which contains the information of all the unmasked tokens. Finally, the speech representations from the speech encoder 10, merged by the fusion network 12, may also be summed with the DAC embeddings.

[0088] The summation of the DAC embeddings and the speech encoder embeddings may then be sent to the masked language model 114, which may comprise L transformer blocks (L= 12 as an example), followed by an MLP layer (not shown). The MLP layer may predict the indices of the masked tokens extracted from the ground truth clean speech, with a vocabulary of 9x1024. Accordingly, the language model 1 1 and the speech encoder 10 may be trained by jointly optimizing a cross-entropy loss 14 and a regression loss 103.

[0089] As a further example, once further conditioned on a target speaker embedding in both the speech encoder and the language model, the system may perform target speaker extraction, which aims at extracting a target speaker’s voice from a distorted mixture containing multiple speakers. Such conditioning may be implemented by Adaptive Layer Normalization (Layemorm) in all the transformer blocks.

[0090] It is noted that the aforementioned example masked language model and loss functions are non-limiting examples for implementing and training the neural network-based language model 11 of the speech restoration system 100a as explicitly illustrated in the present disclosure. Other types of loss functions and other types of language models (e.g., large language models such as an autoregressive language model or non-autoregressive language models, etc.) may be utilized as well and within the scope of the present disclosure. Furthermore, the language model 11 and the speech encoder 10 may also be trained separately in some embodiments.

[0091] Fig. 3 schematically illustrates another example apparatus / system for speech restoration in accordance with embodiments of the present disclosure. Similar to the speech restoration systems 100, 100a, the speech restoration system 100b may be or may be part of a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or a cluster of computers), or a distributed system between a computing device and a server which is suitable for conducting the speech restoration process, and may be operated during a training stage of the speech restoration process, for example, for training a neural network-based language model to restore clean speech from a distorted speech waveform. Also, the speech restoration system 100b may comprise components similar to the speech restoration system 100, such as a speech encoding unit 10, a neural network-based language model 11, a fusion element 12 and a tokenizing unit 13 which perform similar respective functions as the above described speech restoration system 100. In particular, the speech encoding unit 10 may also determine one or more audio features 2- 1 , 2-2, . . . , 2-N associated with the distorted speech waveform for training the speech restoration system 100b.

[0092] The speech restoration system 100b may also be regarded as an example implementation of the speech restoration system 100. In the speech restoration system 100b, semantic representations extracted from target speech (e.g., the ground truth speech 3) may be predicted to introduce semantic knowledge distillation (KD) to the speech encoder 10 for improving the speech intelligibility of the speech restoration system 100b.

[0093] More specifically, the speech encoding unit 10 of the speech restoration system 100b may be further configured to obtain a reference time- frequency representation of a ground truth speech waveform (e.g., the ground truth speech 3) for the distorted speech waveform. Besides, in the speech restoration system 100b, one or more semantic representations 304 associated with the distorted speech waveform may be determined (by e.g., a projection layer 301) based on the one or more audio features 2-1, 2-2, ..., 2-N which has been determined from the input time-frequency representation Iby the speech encoder 10.

[0094] In addition, the speech restoration system 100b may further comprise a pre-trained auxiliary model (e.g., a teacher model) 302 to determine one or more reference semantic representations 305 associated with the ground truth speech waveform 3 from the reference time-frequency representation. Accordingly, the speech encoder 10 of the speech restoration system 100b may be trained based on the one or more semantic representations 304 and the one or more reference semantic representations 305, using an auxiliary loss function 303.

[0095] As an example of implementing the training of the speech restoration system 100b, the one or more semantic representations 304 may be determined from the one or more audio features 2- 1, 2-2, . . ., 2-N by applying a fully connected pooling layer 301 to project the (normalized) audio features to the required dimension. In a more specific example, the one or more semantic representations 304 may comprise one or more pseudo-phonetic labels, and the one or more reference semantic representations 305 may comprise one or more reference pseudo- phonetic labels. Besides, the pre-trained auxiliary model 302 may be implemented by a selfsupervised model. For example, the pre-trained auxiliary model 302 may be a pre-trained HuBERT model.

[0096] Taking the speech encoder 10 of the speech restoration systems 100, 100a as described above as an example, the speech encoder 10 of the speech restoration system 100b may further apply a fully connected (FC) layer to project the normalized features to d dimension, and a stack of self- attention transformer blocks may be used to learn a sequence of latent embeddings. The output from the last transformer block may be used to condition the language model as described above. Compared to the speech restoration system 100a, the speech restoration system 100b may employ an auxiliary loss to train the speech encoder 10, aiming at learning semantic content without transcribed data. Besides, a pre-trained HuBERT-base model may be considered to allow the self-supervised pre-training to encode semantic information by predicting masked pseudo-phonetic labels derived only from audio. For example, given the paired 44.1kHz target speech signal, a sequence of frozen HuBERT targets may be extracted by down-sampling the target speech signal to 16kHz. Since HuBERT has a lower frame rate, an average pooling layer may be appended after the last transformer block in the speech encoder 10 to align the sequences, and an FC layer may output the predicted sequence. Both the pooling layer and the FC layer may form the projection layer 301 as shown in Fig. 3.

[0097] It is noted that the aforementioned example model and architecture are non-limiting examples for taking into account the semantic information for training the proposed speech restoration system 100b as explicitly illustrated in the present disclosure. Other types of models and architectures as well as other approaches for determining the semantic information and training of the system based on the semantic information are feasible and within the scope of the present disclosure.

[0098] Although Fig. 2 and Fig. 3 illustrate two different example implementations of the speech restoration system 100, the skilled person will appreciate that these systems 100, 100a, 100b may be combined to incorporate the training approach based on the semantic information with the speech restoration system 100a. In this way, the speech encoder of the proposed speech restoration systems may be further improved to enhance the speech intelligibility.

[0099] Fig. 4 schematically illustrates an example apparatus / system for speech restoration in accordance with embodiments of the present disclosure. In particular, the apparatus / system 200 may be operated during an inference stage of the speech restoration process, for example, where a trained neural network-based language model (e.g., a language model which has been trained using the speech restoration systems 100, 100a, 100b as illustrated in Figs. 1 to 3) may be employed to restore clean speech from a distorted speech waveform.

[0100] Similar to the systems 100, 100a, 100b, the speech restoration system 200 may be or may be part of a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or a cluster of computers), or a distributed system between a computing device and a server which is suitable for conducting the speech restoration process. In detail, the speech restoration system 200 comprises a speech encoding unit 20, e.g., a neural network-based multi-stage speech encoder, which may be configured to receive or obtain a distorted speech waveform, a machine learning model 21 such as a neural network-based language model (which has been trained for speech restoration), and a fusion element 22 (as a fusion network) coupled between the multi-stage speech encoding unit 20 and the neural network-based language model 21. The speech encoding unit 20 and the fusion element 22 of the speech restoration system 200 may be implemented using architectures / models / approaches similar to the speech encoding unit 10 and the fusion element 12 of the speech restoration systems 100, 100a, 100b. Correspondingly, the speech encoding unit 20 and the fusion element 22 of the speech restoration system 200 may also perform functions similar to the speech encoding unit 10 and the fusion element 12 of the speech restoration systems 100, 100a, 100b, respectively.

[0101] Specifically, the speech encoding unit 20 may be configured to obtain an input timefrequency (temporal-spectral) representation 1 of the distorted speech waveform. Also, the speech encoding unit 20 may determine one or more audio features 2- 1 , 2-2, . . . , 2-N (e.g. , semantic representations, or other non-semantic representations) associated with the distorted speech waveform for determining conditioning information 4 to be provided for the language model 21. The one or more audio features 2-1, 2-2, . . ., 2-N may be determined from the input time-frequency representation using the similar approach as described above for the speech encoding unit 10 of the system 100.

[0102] Since the system 200 may be operated during the inference stage, the neural network-based language model 21 of the system 200 may be configured to generate a plurality of discrete tokens of a codegram 7 corresponding to a clean speech signal. In particular, the plurality of discrete tokens of the codegram 7 may be generated based at least in part on the conditioning information 4 from a fully-masked codegram 5’ containing e.g., (exclusively) a plurality of masked tokens. Specifically, the plurality of discrete tokens of the codegram 7 may be generated based on an iterative sampling procedure.

[0103] Moreover, the system 200 may further comprise a decoding unit 25 configured to generate a restored speech waveform 8 associated with the clean speech signal from the plurality of discrete tokens of the codegram 7. Further, similar to the fusion element 12 of the system 100, the fusion element 22 of the speech restoration system 200 may be configured to generate the conditioning information 4 for the language model 21 based on the one or more audio features 2-1, 2-2, ..., 2-N, for generating the plurality of discrete tokens of the codegram 7.

[0104] It is appreciated that the decoding unit 25 may be integrated with the above described tokenizing unit 13 used for training the language model. This could be achieved by utilizing a pre- trained neural codec (e.g., released by Descript Inc), for example. As an implementation example, a Descript Audio Codec (DAC) based on a convolutional encoder / decoder architecture may be used (as the tokenizing unit 13) to tokenize the audio (associated with the ground truth speech signal 3), where the encoder converts input waveforms sampled at 44. 1 kHz into a latent space, and the decoder converts the latent space back to waveforms to reconstruct the input. For example, a sequence of 9 residual vector quantizers between the encoder and the decoder may discretize each frame of the latent continuous feature vector to 9 discrete tokens, each represented by an integer index ranging from 0 to 1023. Therefore, after tokenization, the reference waveform 3 may become a codegram 5 of size KxT containing discrete values between 0 to 1023, where K=9 is the number of codebooks, and T is the number of frames, whose hop size may be 512 samples (by the design of the convolutional layers in the DAC encoder). It is appreciated that a same hop size of 512 may also be used for generating a spectrogram (e.g., when computing STFT in the speech encoding unit 10) to accurately align the STFT frames one by one with the DAC frames. Accordingly, such kind of tokenizer (e.g., the DAC tokenizer) may also be referred to as a “decoder” according to the present disclosure. Besides, the pre-trained DAC may be kept frozen anytime during training / inference of the language model 1 1, 21.

[0105] It is noted that the aforementioned example DAC is a non-limiting example for implementing the tokenizing unit 13 (and the decoding unit 25) of the proposed speech restoration systems as explicitly illustrated in the present disclosure. Other types of decoders as well as other types of tokenizers and / or other methods for tokenizing audio are also feasible (to replace the above described DAC tokenizer) and within the scope of the present disclosure.

[0106] Preferably, a masked language model may be used for implementing the neural networkbased language model 21, which is to be described in Fig. 5 as below. Additionally, Classifier Free Guidance (CFG) may be applied to mitigate distortions caused by the neural network-based speech restoration systems as described in Fig. 5 below.

[0107] The CFG technique is developed to improve the alignment of generative models to the conditions. According to the present disclosure, the condition to the language model may comprise the fused speech encoder feature(s) (as output by the fusion element 12, 22), which may contain various levels of representations of the input mixture. Additionally, the CFG may be used to reduce distortions to the generated clean speech caused by the neural network processing using the proposed speech restoration systems, since the CFG allows for provision of a controllable parameter to adjust the degree of attachment of the generation process to the mixture.

[0108] Fig. 5 schematically illustrates an example apparatus / system for speech restoration in accordance with embodiments of the present disclosure. Similar to the system architecture shown in Fig. 4 (i.e., the speech restoration system 200), the apparatus / system 200a may be or may be part of a computing device (e.g., a computer, mobile phone, etc.), a server (e.g., a computer or a cluster of computers), or a distributed system between a computing device and a server which is suitable for conducting the speech restoration process, and may be operated during an inference stage of the speech restoration process, for example, where a trained neural network-based language model (e.g., a language model which has been trained using the speech restoration systems 100, 100a, 100b as illustrated in Figs. 1 to 3) may be employed to restore clean speech from a distorted speech waveform. Also, the speech restoration system 200a may comprise components similar to the speech restoration system 200, such as a speech encoding unit 20, a neural network-based language model 21, a fusion element 22 and a decoding unit 25 which perform similar respective functions as the above described speech restoration system 200. The details of these components similar to those in the speech restoration system 200 will not be repeated herein for conciseness.

[0109] The speech restoration system 200a may be regarded as an example implementation of the speech restoration system 200. Similar to the speech restoration system 100a, the speech restoration system 200a may also comprise a masked language model 214, one or more embedding tables 212, and a summation unit 213 which perform similar respective functions as the masked language model 114, the one or more embedding tables 112, and the summation unit 113 of the above described system 100a.

[0110] In order to conduct an iterative sampling procedure for generating the plurality of discrete tokens of the codegram 7, the system 200a may take into account logits of the tokens output from the language model (e.g., logit scores) for sampling output tokens of the language model 214. More specifically, the system 200a may further comprise a logit scoring module 215 configured to compute (for each iteration) a first logit of first predicted tokens generated from the language model 214 using the conditioning information 4 (as determined by the fusion element 22) and, subsequently, computing a second logit of second predicted tokens generated from the language model 214 without the conditioning information 4, or vice versa. For each iteration, the logit scoring module 215 may be further configured to combine the first logit and the second logit using a controllable parameter to obtain a logit with guidance. The controllable parameter may be indicative of a degree of attachment of the generation of the restored speech waveform 8 associated with the clean speech signal to the conditioning information 4. Subsequently, one or more output tokens of the language model 214 generated from a previous iteration may be sampled for re-masking based on the logit with guidance.

[0111] Moreover, the above mentioned fully-masked codegram 5’ (which may contain e.g., (exclusively) a plurality of masked tokens) may be obtained and may then be converted into a plurality of embedding vectors (via the embedding table(s) 212). The plurality of embedding vectors may then be summed (via the summation unit 213), and a summation of the summed embedding vectors with the conditioning information may be provided, based on the logit with guidance, as input to the masked language model 214. Accordingly, the language model 214 may be configured to predict the plurality of masked tokens for predicting / generating the plurality of discrete tokens of the codegram 7 (e.g., by sampling the output token(s) of the language model 214 for re-masking based on the logit with guidance).

[0112] Furthermore, for predicting the plurality of discrete tokens, the iterative sampling procedure performed by the speech restoration system 200a may further comprise determining a predicted accuracy and a predicted gain for each predicted masked token. Accordingly, the plurality of discrete tokens may be generated by comparing a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token with a predefined value. For example, the plurality of discrete tokens may be predicted by retaining a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is greater than or equal to a predefined value. Alternatively, the plurality of discrete tokens may be predicted by discarding a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is smaller than a predefined value. Subsequently, the discarded masked token may be re-predicted by the masked language model 214.

[0113] More specifically, for re-predicting the discarded masked token, the fully-masked codegram 5’ may be updated based on the retained predicted masked token and the discarded masked token. Also, the updated fully-masked codegram may then be converted to a plurality of updated embedding vectors (via the embedding table(s) 212). Furthermore, the logit with guidance may be obtained (e.g., by the logit scoring module 215) based on an output of the masked language model 214 without the conditioning information 4. Subsequently, the plurality of updated embedding vectors may be summed, and a summation of the summed updated embedding vectors with the conditioning information 4 may be provided based on the (obtained) logit with guidance as updated input to the masked language model 214 for the prediction.

[0114] As an example of implementing the CFG in the speech restoration system 200a, predicted tokens generated during the training stage and the inference stage may be taken into account, in order to determine the logit with guidance. Specifically, during training, the fused speech encoder feature may be randomly dropped 10% of the time by replacing it with a learnable embedding, which marks an unconditional language model (as described above with respect to the systems 100, 100a, 100b). During inference, the logits of the tokens (output of the language model) may be computed with and without the fused speech encoder feature (e.g., the conditioning information 4), respectively. The respective logits of the tokens may then be combined according to the following formula to obtain the logits with guidance: lg= (1 + w)lc- wluwhere lcand luare the logits of the predicted tokens with and without the condition, respectively, and w is the positive guidance level. It is appreciated that a larger w may mean more attachment to the fused speech encoder representation (e.g., the combined output feature from the fusion element 12, 22).

[0115] It is noted that distortions to the processed speech caused by neural networks may deteriorate the performance of speech enhancement by a speech restoration system, even though noise can be successfully removed by neural networks. As indicated above, the distortions may include spectral holes, missing high frequency components, changing the speaker voice, cutting off part of a word, etc. In order to mitigate the impact of such distortions in speech restoration, the present disclosure proposes to use classifier free guidance as described above. It is particularly noted that a guidance level w may be defined to determine a degree of attachment of the generation of the restored speech waveform to the conditioning, e.g., more naturalness of the speech may be preserved by a larger guidance w, at the cost of bringing back moderate noise. In practice, the guidance level w may be chosen / selected by both listening and measuring objective quality metrics as a function of w. In the processing during inference, as described above, the language model (e.g., the masked language model 214) may take the fused speech representation from the fusion network 22 (in conjunction with the speech encoder 20) as the condition 4 to the generation process. It may also take a fully-masked codegram 5’, and perform an iterative sampling procedure to gradually generate one or more (or all) of the tokens for the clean speech. The iterative sampling procedure according to the present disclosure may be based on MaskGIT, for example. Besides, classifier free guidance may be used to compute the logits of the predicted tokens as specified above. Finally, the pre-trained DAC decoder 25 as described above may convert the generated clean codegram 7 to a (clean) waveform 8.

[0116] It is noted that the aforementioned example masked language model and the CFG approach are non-limiting examples for inference using the neural network-based language model 21 within the speech restoration system 200a as explicitly illustrated in the present disclosure. Other types of language models (e.g., large language models such as an autoregressive language model or non-autoregressive language models, etc.) and other similar algorithms for determining the logit with guidance may be utilized as well and within the scope of the present disclosure.

[0117] Fig. 6 shows an example flowchart of a method 600 for speech restoration in accordance with embodiments of the disclosure. Method 600 may be implemented in software, hardware, or combinations thereof e.g., at an audio generating device, as or as part of a computing device, a server, or a distributed system between a computing device and a server which is suitable for conducting the speech restoration process, and may be operated during a training stage of the speech restoration process. As shown in Fig. 6, the method 600 comprises processing chains formed by steps S610 through S640 for training a neural network-based language model (e.g., the language model 11, 21 as described above) for restoring clean speech from a distorted speech waveform.

[0118] The method 600 comprises step S610 of obtaining an input time- frequency representation of the distorted speech waveform. The method further comprises step S620 of determining one or more audio features associated with the distorted speech waveform. The one or more audio features may be determined from the input time-frequency representation using a neural network-based multi-stage speech encoder (e.g., the encoding unit 10, 20 as described above). As indicated above, the one or more audio features may also be related to e.g., semantic representations. In addition, the method comprises step S630 of determining conditioning information for the language model based at least in part on the one or more audio features. Besides, the method further comprises step S640 of jointly training the language model and the speech encoder. In particular, the language model may be trained based at least in part on the conditioning information.

[0119] It is further noted that steps S610 through S640 may be performed for each of a plurality of processing cycles of a speech restoration system (e.g., the speech restoration systems 100, 100a, 100b) and do not need to be performed in the order shown in Fig. 6.

[0120] Fig. 7 shows an example flowchart of a method 700 for speech restoration in accordance with embodiments of the disclosure. Method 700 may be implemented in software, hardware, or combinations thereof e.g., at an audio generating device, as or as part of a computing device, a server, or a distributed system between a computing device and a server which is suitable for conducting the speech restoration process, and may be operated during an inference stage of the speech restoration process. As shown in Fig. 7, the method 700 comprises processing chains formed by steps S710 through S750 for using a neural network-based language model (e.g., the language model 11 , 21 as described above) for restoring clean speech from a distorted speech waveform.

[0121] The method 700 comprises step S710 of obtaining an input time- frequency representation of the distorted speech waveform. The method further comprises step S720 of determining one or more audio features associated with the distorted speech waveform. The one or more audio features may be determined from the input time-frequency representation using a neural network-based multi-stage speech encoder (e.g., the encoding unit 10, 20 as described above). As indicated above, the one or more audio features may also be related to e.g., semantic representations. In addition, the method comprises step S730 of determining conditioning information for the language model based at least in part on the one or more audio features. Besides, the method further comprises step S740 of generating a plurality of discrete tokens of a codegram corresponding to a clean speech signal based at least in part on the conditioning information. In particular, the plurality of discrete tokens may be generated from a fully-masked codegram containing a plurality of masked tokens. For example, the plurality of discrete tokens may be generated from a fully-masked codegram containing exclusively a plurality of masked tokens. Moreover, the method also comprises step S750 of generating a restored speech waveform associated with the clean speech signal from the plurality of discrete tokens of the codegram. It is further noted that steps S710 through S750 may be performed for each of a plurality of processing cycles of a speech restoration system (e.g., the speech restoration systems 200, 200a) and do not need to be performed in the order shown in Fig. 7.

[0122] It is understood that the present disclosure likewise relates to an apparatus (e.g., computer- implemented apparatus or apparatus having processing capability in general) for performing or implementing methods and techniques described throughout the present disclosure. For example, this apparatus may relate to an audio decoder. The audio decoder may be a low latency audio decoder, for example, that may be applicable to general audio.

[0123] An example of such apparatus 800 is schematically illustrated in Fig. 8. In particular, the apparatus 800 comprises a processor 810 and a memory 820 coupled to the processor 810. The memory 820 may store instructions for execution by the processor 810. The processor 810 may also receive, among others, suitable input data 830 (e.g., input frames of time-frequency coefficients, etc.), depending on use cases and / or implementations. The processor 810 may be adapted to carry out or implement the methods / techniques described throughout the present disclosure and to generate corresponding output data 840, depending on use cases and / or implementations.

[0124] The apparatus 800 may be configured to perform the functions of the above described components (e.g., a speech encoder, a fusion network / model, a language model, a tokenizer / decoder, etc.) of the proposed speech restoration systems. For example, according to the present disclosure, the apparatus 800 may be configured to determine (e.g., via a speech encoder) features of a speech signal comprising one or more audio distortions, and to fuse the determined features. As indicated above, the fusion of the features may be performed by employing, e.g., a fusion model. In specific implementations, the fusion model may fuse the features using concatenation and / or an attention mechanism. In addition, the apparatus 800 may also be configured to generate discrete tokens of a codegram corresponding to a clean speech signal based on the fused features (e.g., via a masked language model). Accordingly, the apparatus 800 may further be configured to generate a clean speech signal from the discrete tokens (e.g., via a decoder). In some implementations, the decoder may be e.g., an audio tokenizer as described above. The present disclosure likewise relates to corresponding computer programs, computer program products, and computer-readable storage media storing such computer programs or computer program products.

[0125] Aspects of the methods and apparatus / systems described herein may be implemented in an appropriate computer-based audio processing network environment (e.g., server or cloud environment) for processing digital or digitized audio files. Portions of the audio system may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0126] One or more of the components, blocks, processes or other functional components may be implemented through a computer program that controls execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, physical (non-transitory), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.

[0127] Specifically, it should be understood that embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic -based aspects may be implemented in software (e.g., stored on non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and / or application specific integrated circuits (“ASICs”). As such, it should be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components, may be utilized to implement the embodiments. For example, the apparatus (e.g., encoders) described above can include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0128] While one or more implementations have been described by way of example and in terms of the specific embodiments, it is to be understood that one or more implementations are not limited to the disclosed embodiments. To the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art.

[0129] Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0130] Also, it is to be understood that the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted,” “connected,” “supported,” and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings.

[0131] Various Aspects an implementations of the invention may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims.

[0132] EEE Al. A computer- implemented method for speech restoration, including: determining features of a speech signal including one or more audio distortions; fusing the features; generating discrete tokens of a codegram corresponding to a clean speech signal based on the fused features; and generating the clean speech signal from the discrete tokens.

[0133] EEE A2. The method according to EEE Al, wherein the features include intermediate speech representations.

[0134] EEE A3. The method according to EEE Al or EEE A2, wherein the clean speech signal includes one or more reduced audio distortions in comparison with the one or more audio distortions.

[0135] EEE A4. The method according to any one of EEE Al to EEE A3, wherein spectral and / or speech patterns are modeled to determine the features of the speech signal.

[0136] EEE A5. The method according to any one of EEE Al to EEE A4, wherein fusing the features includes concatenation and / or an attention mechanism. EEE A6. The method according to any one of EEE Al to EEE A5, wherein generating the clean speech signal from the discrete tokens includes: generating an audio waveform from the discrete tokens, wherein the clean speech signal includes the audio waveform.

[0137] EEE A7. The method according to any one of EEE Al to EEE A6, further including: performing classifier free guidance to improve speech quality and mitigate over-suppression.

[0138] EEE A8. The method according to any one of EEE Al to EEE A7, wherein the one or more audio distortions include at least one of a spectral hole, a missing high frequency component, missing speech, noise and reverberation exist, clipping and / or a change in speaker voice.

[0139] EEE A9. A system for speech restoration, including: a speech encoder configured to determine features of a speech signal including one or more audio distortions; a fusion model configured to fuse the features; a language model configured to generate discrete tokens of a codegram corresponding to a clean speech signal based on the fused features; and a decoder configured to generate the clean speech signal from the discrete tokens.

[0140] EEE A 10. The system according to EEE A9, wherein the speech encoder and the language model are jointly trained.

[0141] EEE Al l. The system according to EEE A9 or EEE A 10, wherein the features include intermediate speech representations.

[0142] EEE Al 2. The system according to any one of EEE A9 to EEE Al 1, wherein the clean speech signal includes one or more reduced audio distortions in comparison with the one or more audio distortions.

[0143] EEE Al 3. The system according to any one of EEE A9 to EEE Al 2, wherein the speech encoder includes: one or more multilayer perceptron; and one or more transformer blocks, wherein the one or more transformer blocks are configured to determine the features of the speech signal.

[0144] EEE A 14. The system according to EEE A 13, wherein the one or more transformer blocks model spectral and / or speech patterns of the speech signal to determine the features.

[0145] EEE Al 5. The system according to any one of EEE A9 to EEE A 14, wherein the speech encoder is trained to minimize a regression loss function or a cross-entropy loss function, and the language model is trained to minimize the regression loss function or the cross entropy loss function.

[0146] EEE Al 6. The system according to any one of EEE A9 to EEE A 15, wherein the fusion model is configured to fuse the features via concatenation and / or via an attention mechanism.

[0147] EEE A17. The system according to any one of EEE A9 to EEE A16, wherein the decoder is configured to generate an audio waveform from the discrete tokens, wherein the clean speech signal includes the audio waveform.

[0148] EEE Al 8. The system according to any one of EEE A9 to EEE A 17, further configured to improve speech quality and mitigate over-suppression via classifier free guidance.

[0149] EEE A19. The system according to any one of EEE A9 to EEE Al 8, wherein the one or more audio distortions include at least one of a spectral hole, a missing high frequency component, missing speech, noise and reverberation exist, clipping and / or a change in speaker voice.

[0150] EEE A20. The system according to any one of EEE A9 to EEE A 19, wherein the language model includes a non-autoregressive language model.

[0151] EEE A21. The system according to any one of EEE A9 to EEE A20, wherein the language model is a masked language model including a MaskGIT model.

[0152] EEE A22. The system according to any one of EEE A9 to EEE A 19, wherein the language model includes an autoregressive language model.

[0153] EEE A23. An apparatus for speech restoration, including a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of EEE Al to EEE A8.

[0154] EEE A24. A computer program including instructions that, when executed by a processor, cause the processor to perform all steps of the method according to any one of EEE Al to EEE A8.

[0155] EEE A25. A computer-readable storage medium storing the computer program according to EEE A24.

[0156] EEE B 1. A computer-implemented method of using a neural network-based language model for restoring clean speech from a distorted speech waveform, the method comprising: obtaining an input time-frequency representation of the distorted speech waveform; determining, from the input time-frequency representation, one or more audio features associated with the distorted speech waveform using a neural network-based multi-stage speech encoder; determining conditioning information for the language model based at least in part on the one or more audio features; generating, from a fully-masked codegram containing a plurality of masked tokens, a plurality of discrete tokens of a codegram corresponding to a clean speech signal based at least in part on the conditioning information; and generating a restored speech waveform associated with the clean speech signal from the plurality of discrete tokens of the codegram.

[0157] EEE B2. The method according to EEE Bl, wherein the plurality of discrete tokens of the codegram is generated based on an iterative sampling procedure comprising, for each iteration: computing a first logit of first predicted tokens generated from the language model using the conditioning information and, subsequently, computing a second logit of second predicted tokens generated from the language model without the conditioning information, or vice versa; combining the first logit and the second logit using a controllable parameter to obtain a logit with guidance; and sampling one or more output tokens of the language model generated from a previous iteration for re-masking based on the logit with guidance.

[0158] EEE B3. The method according to EEE B2, wherein the controllable parameter is indicative of a degree of attachment of the generation of the restored speech waveform associated with the clean speech signal to the conditioning information.

[0159] EEE B4. The method according to any one of EEEs Bl to B3, wherein the language model comprises an autoregressive language model.

[0160] EEE B5. The method according to EEE B3, wherein the language model comprises a masked language model, the method further comprising: obtaining the fully-masked codegram containing a plurality of masked tokens; converting the fully-masked codegram to a plurality of embedding vectors; summing the plurality of embedding vectors; providing, based on the logit with guidance, a summation of the summed embedding vectors with the conditioning information as input to the masked language model; and predicting, by the language model, the plurality of masked tokens for predicting the plurality of discrete tokens of the codegram.

[0161] EEE B6. The method according to EEE B5, wherein the iterative sampling procedure further comprises determining a predicted accuracy and a predicted gain for each predicted masked token, and wherein predicting the plurality of discrete tokens comprises: retaining a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is greater than or equal to a predefined value, or discarding a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is smaller than a predefined value; and re-predicting the discarded masked token by the masked language model.

[0162] EEE B7. The method according to EEE B6, wherein re-predicting the discarded masked token comprises: updating the fully-masked codegram based on the retained predicted masked token and the discarded masked token; converting the updated fully-masked codegram to a plurality of updated embedding vectors; obtaining the logit with guidance based on an output of the masked language model without the conditioning information; summing the plurality of updated embedding vectors; providing, based on the logit with guidance, a summation of the summed updated embedding vectors with the conditioning information as updated input to the masked language model for prediction. EEE B8. The method according to any one of EEEs Bl to B7, wherein the one or more audio features are determined based on a plurality of intermediate speech representations of the input time-frequency representation generated within the speech encoder.

[0163] EEE B9. The method according to EEE B8, wherein the plurality of intermediate speech representations is generated by applying a plurality of successive transformer layers each generating a respective intermediate speech representation.

[0164] EEE BIO. The method according to EEE B9, further comprising, for determining the plurality of intermediate speech representations, transforming, by a second multi-layer perceptron, a frame size of the input time-frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers.

[0165] EEE Bl 1. The method according to EEE BIO, further comprising combining a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model.

[0166] EEE Bl 2. The method according to EEE Bl 1, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multi-layer perceptron and the respective transformer layers as a feature vector and applying a learnable projection to the feature vector using a linear layer.

[0167] EEE Bl 3. The method according to EEE Bl 2, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by summing one or more features from the further multi-layer perceptron and the respective transformer layers with a corresponding weight coefficient determined by applying an attention algorithm.

[0168] EEE Bl 4. A computer- implemented method of training a neural network-based language model for restoring clean speech from a distorted speech waveform, the method comprising: obtaining an input time-frequency representation of the distorted speech waveform; determining, from the input time-frequency representation, one or more audio features associated with the distorted speech waveform using a neural network-based multi-stage speech encoder; determining conditioning information for the language model based at least in part on the one or more audio features; and jointly training the language model and the speech encoder, wherein the language model is trained based at least in part on the conditioning information,

[0169] EEE B15. The method according to EEE B14, further comprising: obtaining a reference time-frequency representation of a ground truth speech waveform for the distorted speech waveform; determining one or more semantic representations associated with the distorted speech waveform based on the one or more audio features; and determining, from the reference time-frequency representation, one or more reference semantic representations associated with the ground truth speech waveform using a pretrained auxiliary model, wherein the speech encoder is trained based on the one or more semantic representations and the one or more reference semantic representations, using an auxiliary loss function.

[0170] EEE Bl 6. The method according to EEE Bl 5, wherein determining the one or more semantic representations from the one or more audio features comprises applying a fully connected pooling layer.

[0171] EEE Bl 7. The method according to EEE B15 or B16, wherein the one or more semantic representations comprise one or more pseudo-phonetic labels, and wherein the one or more reference semantic representations comprise one or more reference pseudo-phonetic labels.

[0172] EEE Bl 8. The method according to any one of EEEs B 15 to B 17, wherein the pre-trained auxiliary model is a self-supervised model.

[0173] EEE Bl 9. The method according to any one of EEEs B15 to B18, wherein the pre-trained auxiliary model is a pre-trained HuBERT model. EEE B20. The method according to any one of EEEs B 14 to B 19, further comprising: obtaining a reference time-frequency representation of a ground truth speech waveform for the distorted speech waveform, wherein the speech encoder is trained based on the reference time-frequency representation and a predicted time-frequency representation of a clean speech waveform predicted by the speech encoder, using a regression loss.

[0174] EEE B21. The method according to EEE B20, further comprising applying a multi-layer perceptron to generate the predicted time-frequency representation.

[0175] EEE B22. The method according to EEE B20 or B21, wherein training the speech encoder comprises: minimizing a difference between the predicted time-frequency representation and the reference time-frequency representation by adapting one or more parameters of the multistage speech encoder based on a regression loss function.

[0176] EEE B23. The method according to any one of EEE B14 or EEEs B20 to B22, wherein the one or more audio features are determined based on a plurality of intermediate speech representations of the input time-frequency representation generated within the speech encoder.

[0177] EEE B24. The method according to EEE B23, wherein the plurality of intermediate speech representations is generated by applying a plurality of successive transformer layers each generating a respective intermediate speech representation.

[0178] EEE B25. The method according to EEE B24, further comprising, for determining the plurality of intermediate speech representations, transforming, by a second multi-layer perceptron, a frame size of the input time-frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers. EEE B26. The method according to EEE B25, further comprising combining a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model.

[0179] EEE B27. The method according to EEE B26, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multi-layer perceptron and the respective transformer layers as a feature vector and applying a learnable projection to the feature vector using a linear layer.

[0180] EEE B28. The method according to EEE B26, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by summing one or more features from the second multi-layer perceptron and the respective transformer layers with a corresponding weight coefficient determined by applying an attention algorithm.

[0181] EEE B29. The method according to any one of EEEs B14 to B28, further comprising: randomly providing the conditioning information to the language model for conditioned training of the language model; and replacing a portion of the conditioning information with one or more learnable embedding parameters as unconditional input for training the language model without condition, wherein training the language model is based further on the unconditional input comprising the one or more learnable embedding parameters.

[0182] EEE B30. The method according to EEE B29, wherein a number of training cycles for conditioned training of the language model is less than a total number of training cycles for training of the language model, and a number of training cycles for training the language model without condition is given by a predetermined ratio.

[0183] EEE B31. The method according to any one of EEEs B14 to B30, further comprising tokenizing a reference waveform associated with a ground truth speech signal to convert the reference waveform into a codegram of a predefined size containing discrete tokens for training the language model.

[0184] EEE B32 The method according to any one of EEEs B14 to B31, wherein the language model is trained based on a cross-entropy loss function.

[0185] EEE B33. The method according to any one of EEEs B14 to B32, wherein the language model comprises an autoregressive language model.

[0186] EEE B34. The method according to any one of EEEs B14 to B33, wherein the language model comprises a masked language model.

[0187] EEE B35. The method according to EEE B34, further comprising randomly masking one or more discrete tokens in the codegram to generate a partially-masked codegram, converting the partially-masked codegram to a plurality of embedding vectors, summing the plurality of embedding vectors, and providing a summation of the summed embedding vectors with the conditioning information as training input to the masked language model.

[0188] EEE B36. The method according to EEE B35, wherein the language model comprises a plurality of transformer blocks followed by a multi-layer perceptron, the method further comprising, for training the language model: extracting a plurality of masked tokens from the ground truth speech signal; estimating, by the multi-layer perceptron, a plurality of predicted indices corresponding to the masked tokens; determining a difference between the predicted indices and actual indices of the masked tokens; and minimizing the difference between the predicted indices and the actual indices of the masked tokens based on a cross-entropy loss function.

[0189] EEE B37. A system for speech restoration, comprising: a neural network-based multi-stage speech encoding unit configured to: obtain an input time-frequency representation of a distorted speech waveform, and determine, from the input time-frequency representation, one or more audio features associated with the distorted speech waveform; a neural network-based language model configured to generate a plurality of discrete tokens of a codegram corresponding to a clean speech signal; a decoding unit configured to generate a restored speech waveform associated with the clean speech signal from the plurality of discrete tokens of the codegram; and a fusion element configured to generate conditioning information for the language model based on the one or more audio features, wherein the plurality of discrete tokens of the codegram is generated based at least in part on the conditioning information.

[0190] EEE B38. The system according to EEE B37, wherein the speech encoding unit comprises a multi-layer perceptron configured to generate a predicted time-frequency representation of a clean speech waveform based on a plurality of intermediate speech representations of the input time-frequency representation.

[0191] EEE B39. The system according to EEE B38, wherein the speech encoding unit is further configured to: determine one or more semantic representations based on the one or more audio features.

[0192] EEE B40. The system according to EEE B38 or B39, wherein the speech encoding unit further comprises a plurality of successive transformer layers each configured to generate a respective intermediate speech representation.

[0193] EEE B41. The system according to EEE B40, wherein the speech encoding unit further comprises a second multi-layer perceptron configured to transform a frame size of the input time-frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers.

[0194] EEE B42. The system according to EEE B41 , wherein the fusion element is configured to combine a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model. EEE B43. The system according to any one of EEEs B37 to B42, further comprising a tokenization unit configured to tokenize a reference waveform associated with a ground truth speech signal to convert the reference waveform into a reference codegram of a predefined size containing reference discrete tokens for training the language model.

[0195] EEE B44. The system according to any one of EEEs B37 to B43, wherein the speech encoding unit and the language model are jointly trained based on a regression loss function and / or a cross-entropy function.

[0196] EEE B45. The system according to any one of EEEs B37 to B44, wherein the language model comprises an autoregressive language model, or a masked language model.

[0197] EEE B46. The system according to EEE B43, wherein the language model comprises a masked language model, the system further comprising a masking unit configured to randomly mask one or more reference discrete tokens in the reference codegram to generate a partially-masked codegram, an embedding unit configured to convert the partially-masked codegram to a plurality of embedding vectors, and a summing unit configured to sum the plurality of embedding vectors and to provide a summation of the summed embedding vectors with the conditioning information as training input to the masked language model.

[0198] EEE B47. The system according to EEE B46, wherein the masked language model comprises a plurality of transformer blocks followed by a multi-layer perceptron configured to, for training the masked language model: extract a plurality of masked tokens from the ground truth speech signal; estimate a plurality of predicted indices corresponding to the masked tokens; determine a difference between the predicted indices and actual indices of the masked tokens; and minimize the difference between the predicted indices and the actual indices of the masked tokens based on a cross-entropy loss function.

[0199] EEE B48. The system according to any one of EEEs B37 to B47, further comprising a logit estimation unit configured to: compute a first logit of first predicted tokens generated from the language model using the conditioning information and, subsequently, compute a second logit of second predicted tokens generated from the language model without the conditioning information, or vice versa; and combine the first logit and the second logit using a controllable parameter to obtain a logit with guidance, the controllable parameter indicating of a degree of attachment of the generation of the restored speech waveform associated with the clean speech signal to the conditioning information, wherein the plurality of discrete tokens of the codegram is generated based on an iterative sampling procedure by classifier free guidance (CFG) comprising sampling, for each iteration, one or more output tokens of the language model generated from a previous iteration for re-masking based on the logit with guidance.

[0200] EEE B49. An apparatus for speech restoration, comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of EEEs Bl to B36.

[0201] EEE B50. A computer program comprising instructions that, when executed by a computing device, cause the computing device to perform all steps of the method according to any one of EEEs Bl to B36.

[0202] EEE B51. A computer-readable storage medium storing the computer program according to EEE B50.

Claims

CLAIMS1. A computer-implemented method of using a neural network-based language model for restoring clean speech from a distorted speech waveform, the method comprising: obtaining an input time-frequency representation of the distorted speech waveform; determining, from the input time-frequency representation, one or more audio features associated with the distorted speech waveform using a neural network-based multi-stage speech encoder; determining conditioning information for the language model based at least in part on the one or more audio features; generating, from a fully-masked codegram containing a plurality of masked tokens, a plurality of discrete tokens of a codegram corresponding to a clean speech signal based at least in part on the conditioning information; and generating a restored speech waveform associated with the clean speech signal from the plurality of discrete tokens of the codegram.

2. The method according to claim 1 , wherein the plurality of discrete tokens of the codegram is generated based on an iterative sampling procedure comprising, for each iteration: computing a first logit of first predicted tokens generated from the language model using the conditioning information and, subsequently, computing a second logit of second predicted tokens generated from the language model without the conditioning information, or vice versa; combining the first logit and the second logit using a controllable parameter to obtain a logit with guidance; and sampling one or more output tokens of the language model generated from a previous iteration for re-masking based on the logit with guidance.

3. The method according to claim 2, wherein the controllable parameter is indicative of a degree of attachment of the generation of the restored speech waveform associated with the clean speech signal to the conditioning information.

4. The method according to any one of claims 1 to 3, wherein the language model comprises an autoregressive language model.

5. The method according to claim 3, wherein the language model comprises a masked language model, the method further comprising: obtaining the fully-masked codegram containing a plurality of masked tokens; converting the fully-masked codegram to a plurality of embedding vectors; summing the plurality of embedding vectors; providing, based on the logit with guidance, a summation of the summed embedding vectors with the conditioning information as input to the masked language model; and predicting, by the language model, the plurality of masked tokens for predicting the plurality of discrete tokens of the codegram.

6. The method according to claim 5, wherein the iterative sampling procedure further comprises determining a predicted accuracy and a predicted gain for each predicted masked token, and wherein predicting the plurality of discrete tokens comprises: retaining a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is greater than or equal to a predefined value, or discarding a predicted masked token if a corresponding predicted accuracy and / or a corresponding predicted gain associated with the predicted masked token is smaller than a predefined value; and re-predicting the discarded masked token by the masked language model.

7. The method according to claim 6, wherein re-predicting the discarded masked token comprises: updating the fully-masked codegram based on the retained predicted masked token and the discarded masked token; converting the updated fully-masked codegram to a plurality of updated embedding vectors; obtaining the logit with guidance based on an output of the masked language model without the conditioning information; summing the plurality of updated embedding vectors;providing, based on the logit with guidance, a summation of the summed updated embedding vectors with the conditioning information as updated input to the masked language model for prediction.

8. The method according to any one of claims 1 to 7, wherein the one or more audio features are determined based on a plurality of intermediate speech representations of the input time-frequency representation generated within the speech encoder.

9. The method according to claim 8, wherein the plurality of intermediate speech representations is generated by applying a plurality of successive transformer layers each generating a respective intermediate speech representation.

10. The method according to claim 9, further comprising, for determining the plurality of intermediate speech representations, transforming, by a second multi-layer perceptron, a frame size of the input time-frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers.

11. The method according to claim 10, further comprising combining a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model.

12. The method according to claim 11, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multi-layer perceptron and the respective transformer layers as a feature vector and applying a learnable projection to the feature vector using a linear layer.

13. The method according to claim 12, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by summing one or more features from the further multi-layer perceptron and the respective transformer layers with a corresponding weight coefficient determined by applying an attention algorithm.

14. A computer-implemented method of training a neural network-based language model for restoring clean speech from a distorted speech waveform, the method comprising: obtaining an input time-frequency representation of the distorted speech waveform; determining, from the input time-frequency representation, one or more audio features associated with the distorted speech waveform using a neural network-based multi-stage speech encoder; determining conditioning information for the language model based at least in part on the one or more audio features; and jointly training the language model and the speech encoder, wherein the language model is trained based at least in part on the conditioning information,15. The method according to claim 14, further comprising: obtaining a reference time-frequency representation of a ground truth speech waveform for the distorted speech waveform; determining one or more semantic representations associated with the distorted speech waveform based on the one or more audio features; and determining, from the reference time-frequency representation, one or more reference semantic representations associated with the ground truth speech waveform using a pretrained auxiliary model, wherein the speech encoder is trained based on the one or more semantic representations and the one or more reference semantic representations, using an auxiliary loss function.

16. The method according to claim 15, wherein determining the one or more semantic representations from the one or more audio features comprises applying a fully connected pooling layer.

17. The method according to claim 15 or 16, wherein the one or more semantic representations comprise one or more pseudo-phonetic labels, and wherein the one or more reference semantic representations comprise one or more reference pseudo-phonetic labels.

18. The method according to any one of claims 15 to 17, wherein the pre-trained auxiliary model is a self-supervised model.

19. The method according to any one of claims 15 to 18, wherein the pre-trained auxiliary model is a pre-trained HuBERT model.

20. The method according to any one of claims 14 to 19, further comprising: obtaining a reference time-frequency representation of a ground truth speech waveform for the distorted speech waveform, wherein the speech encoder is trained based on the reference time-frequency representation and a predicted time-frequency representation of a clean speech waveform predicted by the speech encoder, using a regression loss.

21. The method according to claim 20, further comprising applying a multi-layer perceptron to generate the predicted time-frequency representation.

22. The method according to claim 20 or 21, wherein training the speech encoder comprises: minimizing a difference between the predicted time-frequency representation and the reference time-frequency representation by adapting one or more parameters of the multistage speech encoder based on a regression loss function.

23. The method according to any one of claim 14 or claims 20 to 22, wherein the one or more audio features are determined based on a plurality of intermediate speech representations of the input time-frequency representation generated within the speech encoder.

24. The method according to claim 23, wherein the plurality of intermediate speech representations is generated by applying a plurality of successive transformer layers each generating a respective intermediate speech representation.

25. The method according to claim 24, further comprising, for determining the plurality of intermediate speech representations, transforming, by a second multi-layerperceptron, a frame size of the input time-frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers.

26. The method according to claim 25, further comprising combining a further intermediate speech representation from the second multi-layer perceptron and the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model.

27. The method according to claim 26, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by concatenating one or more features from the second multi-layer perceptron and the respective transformer layers as a feature vector and applying a learnable projection to the feature vector using a linear layer.

28. The method according to claim 26, wherein the further intermediate speech representation and the plurality of intermediate speech representations are combined by summing one or more features from the second multi-layer perceptron and the respective transformer layers with a corresponding weight coefficient determined by applying an attention algorithm.

29. The method according to any one of claims 14 to 28, further comprising: randomly providing the conditioning information to the language model for conditioned training of the language model; and replacing a portion of the conditioning information with one or more learnable embedding parameters as unconditional input for training the language model without condition, wherein training the language model is based further on the unconditional input comprising the one or more learnable embedding parameters.

30. The method according to claim 29, wherein a number of training cycles for conditioned training of the language model is less than a total number of training cycles for training of the language model, and a number of training cycles for training the language model without condition is given by a predetermined ratio.

31. The method according to any one of claims 14 to 30, further comprising tokenizing a reference waveform associated with a ground truth speech signal to convert the reference waveform into a codegram of a predefined size containing discrete tokens for training the language model.32 The method according to any one of claims 14 to 31, wherein the language model is trained based on a cross-entropy loss function.

33. The method according to any one of claims 14 to 32, wherein the language model comprises an autoregressive language model.

34. The method according to any one of claims 14 to 33, wherein the language model comprises a masked language model.

35. The method according to claim 34, further comprising randomly masking one or more discrete tokens in the codegram to generate a partially-masked codegram, converting the partially-masked codegram to a plurality of embedding vectors, summing the plurality of embedding vectors, and providing a summation of the summed embedding vectors with the conditioning information as training input to the masked language model.

36. The method according to claim 35, wherein the language model comprises a plurality of transformer blocks followed by a multi-layer perceptron, the method further comprising, for training the language model: extracting a plurality of masked tokens from the ground truth speech signal; estimating, by the multi-layer perceptron, a plurality of predicted indices corresponding to the masked tokens; determining a difference between the predicted indices and actual indices of the masked tokens; and minimizing the difference between the predicted indices and the actual indices of the masked tokens based on a cross-entropy loss function.

37. A system for speech restoration, comprising:a neural network-based multi-stage speech encoding unit configured to: obtain an input time-frequency representation of a distorted speech waveform, and determine, from the input time-frequency representation, one or more audio features associated with the distorted speech waveform; a neural network-based language model configured to generate a plurality of discrete tokens of a codegram corresponding to a clean speech signal; a decoding unit configured to generate a restored speech waveform associated with the clean speech signal from the plurality of discrete tokens of the codegram; and a fusion element configured to generate conditioning information for the language model based on the one or more audio features, wherein the plurality of discrete tokens of the codegram is generated based at least in part on the conditioning information.

38. The system according to claim 37, wherein the speech encoding unit comprises a multi-layer perceptron configured to generate a predicted time-frequency representation of a clean speech waveform based on a plurality of intermediate speech representations of the input time-frequency representation.

39. The system according to claim 38, wherein the speech encoding unit is further configured to: determine one or more semantic representations based on the one or more audio features.

40. The system according to claim 38 or 39, wherein the speech encoding unit further comprises a plurality of successive transformer layers each configured to generate a respective intermediate speech representation.

41. The system according to claim 40, wherein the speech encoding unit further comprises a second multi-layer perceptron configured to transform a frame size of the input time-frequency representation to a transformer layer size for passing the input time-frequency representation to the plurality of successive transformer layers.

42. The system according to claim 41, wherein the fusion element is configured to combine a further intermediate speech representation from the second multi-layer perceptronand the plurality of intermediate speech representations from the plurality of successive transformer layers as the conditioning information to be provided to the language model.

43. The system according to any one of claims 37 to 42, further comprising a tokenization unit configured to tokenize a reference waveform associated with a ground truth speech signal to convert the reference waveform into a reference codegram of a predefined size containing reference discrete tokens for training the language model.

44. The system according to any one of claims 37 to 43, wherein the speech encoding unit and the language model are jointly trained based on a regression loss function and / or a cross-entropy function.

45. The system according to any one of claims 37 to 44, wherein the language model comprises an autoregressive language model, or a masked language model.

46. The system according to claim 43, wherein the language model comprises a masked language model, the system further comprising a masking unit configured to randomly mask one or more reference discrete tokens in the reference codegram to generate a partially-masked codegram, an embedding unit configured to convert the partially-masked codegram to a plurality of embedding vectors, and a summing unit configured to sum the plurality of embedding vectors and to provide a summation of the summed embedding vectors with the conditioning information as training input to the masked language model.

47. The system according to claim 46, wherein the masked language model comprises a plurality of transformer blocks followed by a multi-layer perceptron configured to, for training the masked language model: extract a plurality of masked tokens from the ground truth speech signal; estimate a plurality of predicted indices corresponding to the masked tokens; determine a difference between the predicted indices and actual indices of the masked tokens; and minimize the difference between the predicted indices and the actual indices of the masked tokens based on a cross-entropy loss function.

48. The system according to any one of claims 37 to 47, further comprising a logit estimation unit configured to: compute a first logit of first predicted tokens generated from the language model using the conditioning information and, subsequently, compute a second logit of second predicted tokens generated from the language model without the conditioning information, or vice versa; and combine the first logit and the second logit using a controllable parameter to obtain a logit with guidance, the controllable parameter indicating of a degree of attachment of the generation of the restored speech waveform associated with the clean speech signal to the conditioning information, wherein the plurality of discrete tokens of the codegram is generated based on an iterative sampling procedure by classifier free guidance (CFG) comprising sampling, for each iteration, one or more output tokens of the language model generated from a previous iteration for re-masking based on the logit with guidance.

49. An apparatus for speech restoration, comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of claims 1 to 36.

50. A computer program comprising instructions that, when executed by a computing device, cause the computing device to perform all steps of the method according to any one of claims 1 to 36.

51. A computer-readable storage medium storing the computer program according to claim 50.

Images